[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 487822841 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002340] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007303] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010137] LSM: Security Framework initializing [ 0.011051] Yama: becoming mindful. [ 0.012033] SELinux: Initializing. [ 0.013066] *** VALIDATE selinux *** [ 0.020740] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024824] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027116] *** VALIDATE tmpfs *** [ 0.029286] *** VALIDATE proc *** [ 0.030220] *** VALIDATE cgroup *** [ 0.031009] *** VALIDATE cgroup2 *** [ 0.032273] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033152] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035029] Spectre V2 : User space: Vulnerable [ 0.036008] Speculative Store Bypass: Vulnerable [ 0.038670] debug: unmapping init [mem 0xffffffffb3459000-0xffffffffb3460fff] [ 0.040223] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041667] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042023] ... version: 2 [ 0.043013] ... bit width: 48 [ 0.044013] ... generic registers: 4 [ 0.045013] ... value mask: 0000ffffffffffff [ 0.046013] ... max period: 00007fffffffffff [ 0.047013] ... fixed-purpose events: 3 [ 0.048011] ... event mask: 000000070000000f [ 0.049263] rcu: Hierarchical SRCU implementation. [ 0.051455] smp: Bringing up secondary CPUs ... [ 0.052584] x86: Booting SMP configuration: [ 0.053038] .... node #0, CPUs: #1 #2 #3 [ 0.057013] smp: Brought up 1 node, 4 CPUs [ 0.059014] smpboot: Max logical packages: 1 [ 0.060015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.127239] node 0 deferred pages initialised in 65ms [ 0.131008] devtmpfs: initialized [ 0.132228] x86/mm: Memory block size: 128MB [ 0.134716] gcov: version magic: 0x41383552 [ 0.135623] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.140184] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.143453] pinctrl core: initialized pinctrl subsystem [ 0.146248] [ 0.146841] ************************************************************* [ 0.149031] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152021] ** ** [ 0.155018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158028] ** ** [ 0.161020] ** This means that this kernel is built to expose internal ** [ 0.164020] ** IOMMU data structures, which may compromise security on ** [ 0.166029] ** your system. ** [ 0.169034] ** ** [ 0.172021] ** If you see this message and you are not debugging the ** [ 0.175018] ** kernel, report this immediately to your vendor! ** [ 0.178023] ** ** [ 0.180017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.183020] ************************************************************* [ 0.186907] NET: Registered protocol family 16 [ 0.188512] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.192093] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.195091] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199162] cpuidle: using governor menu [ 0.200867] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.204567] PCI: Using configuration type 1 for base access [ 0.207127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.217076] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.220040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.223037] cryptd: max_cpu_qlen set to 1000 [ 0.225309] ACPI: Added _OSI(Module Device) [ 0.228024] ACPI: Added _OSI(Processor Device) [ 0.230018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.232015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.237094] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.243150] ACPI: Interpreter enabled [ 0.244064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.246014] ACPI: Using IOAPIC for interrupt routing [ 0.248112] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.252413] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.262995] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.266066] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.269029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.273117] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.279561] acpiphp: Slot [2] registered [ 0.281140] acpiphp: Slot [5] registered [ 0.282139] acpiphp: Slot [6] registered [ 0.284256] acpiphp: Slot [7] registered [ 0.286121] acpiphp: Slot [8] registered [ 0.288200] acpiphp: Slot [9] registered [ 0.289098] acpiphp: Slot [10] registered [ 0.291139] acpiphp: Slot [3] registered [ 0.293107] acpiphp: Slot [4] registered [ 0.295132] acpiphp: Slot [11] registered [ 0.296211] acpiphp: Slot [12] registered [ 0.298101] acpiphp: Slot [13] registered [ 0.300105] acpiphp: Slot [14] registered [ 0.302167] acpiphp: Slot [15] registered [ 0.303147] acpiphp: Slot [16] registered [ 0.305108] acpiphp: Slot [17] registered [ 0.306139] acpiphp: Slot [18] registered [ 0.308076] acpiphp: Slot [19] registered [ 0.309152] acpiphp: Slot [20] registered [ 0.311103] acpiphp: Slot [21] registered [ 0.313134] acpiphp: Slot [22] registered [ 0.314103] acpiphp: Slot [23] registered [ 0.316145] acpiphp: Slot [24] registered [ 0.318152] acpiphp: Slot [25] registered [ 0.319142] acpiphp: Slot [26] registered [ 0.322195] acpiphp: Slot [27] registered [ 0.324157] acpiphp: Slot [28] registered [ 0.326150] acpiphp: Slot [29] registered [ 0.327130] acpiphp: Slot [30] registered [ 0.329155] acpiphp: Slot [31] registered [ 0.331094] PCI host bridge to bus 0000:00 [ 0.332028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.335033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.338033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.340033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.343050] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.346049] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.351183] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.355092] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.358280] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.366015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.372123] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.374020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.376019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.379015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.382571] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.386125] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.389064] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.392913] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.398023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.412022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.418901] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.425311] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.433017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.440017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.459023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.470515] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.478021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.485022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.505022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.517469] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.525021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.532019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.551025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.560872] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.568024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.579021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.600020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.617185] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.629018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.633015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.648015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.659361] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.667021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.675022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.697017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.715053] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.717395] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.719377] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.722388] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.724216] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.729069] iommu: Default domain type: Passthrough [ 0.731408] SCSI subsystem initialized [ 0.733143] ACPI: bus type USB registered [ 0.734098] usbcore: registered new interface driver usbfs [ 0.735073] usbcore: registered new interface driver hub [ 0.737117] usbcore: registered new device driver usb [ 0.739160] pps_core: LinuxPPS API ver. 1 registered [ 0.741013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.743065] PTP clock support registered [ 0.747131] EDAC MC: Ver: 3.0.0 [ 0.749534] PCI: Using ACPI for IRQ routing [ 0.751963] NetLabel: Initializing [ 0.754015] NetLabel: domain hash size = 128 [ 0.755012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.757105] NetLabel: unlabeled traffic allowed by default [ 0.760155] vgaarb: loaded [ 0.762431] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.763014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.770157] clocksource: Switched to clocksource kvm-clock [ 0.890110] VFS: Disk quotas dquot_6.6.0 [ 0.891907] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.894821] *** VALIDATE ramfs *** [ 0.896368] *** VALIDATE hugetlbfs *** [ 0.898230] pnp: PnP ACPI init [ 0.900917] pnp: PnP ACPI: found 6 devices [ 0.933435] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.936824] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.939408] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.941761] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.944520] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.947161] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.949992] NET: Registered protocol family 2 [ 0.952770] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.958698] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.962688] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.968979] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.972852] TCP: Hash tables configured (established 65536 bind 65536) [ 0.976326] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.979926] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.982749] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.985786] NET: Registered protocol family 1 [ 0.988744] RPC: Registered named UNIX socket transport module. [ 0.991536] RPC: Registered udp transport module. [ 0.993663] RPC: Registered tcp transport module. [ 0.995765] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.998661] NET: Registered protocol family 44 [ 1.000942] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.003749] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.006480] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.009736] PCI: CLS 0 bytes, default 64 [ 1.012515] Unpacking initramfs... [ 2.439393] debug: unmapping init [mem 0xffff96847cc54000-0xffff96847ffbffff] [ 2.443898] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.446039] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.448987] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.941172] Initialise system trusted keyrings [ 2.943018] Key type blacklist registered [ 2.945092] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.953827] zbud: loaded [ 2.956658] *** VALIDATE nfs *** [ 2.958059] *** VALIDATE nfs4 *** [ 2.959473] pstore: using deflate compression [ 2.963116] Platform Keyring initialized [ 3.075683] NET: Registered protocol family 38 [ 3.077419] Key type asymmetric registered [ 3.079070] Asymmetric key parser 'x509' registered [ 3.081455] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.085071] io scheduler mq-deadline registered [ 3.086611] io scheduler kyber registered [ 3.088173] io scheduler bfq registered [ 3.090531] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.093631] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.096290] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.099114] ACPI: Power Button [PWRF] [ 3.104996] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.111777] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.127627] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.135644] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.158183] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.186875] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.215818] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.221663] Non-volatile memory driver v1.3 [ 3.223555] Linux agpgart interface v0.103 [ 3.257829] virtio_blk virtio1: [vda] 150424 512-byte logical blocks (77.0 MB/73.4 MiB) [ 3.262025] vda: detected capacity change from 0 to 77017088 [ 3.282302] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.285432] vdb: detected capacity change from 0 to 1073741824 [ 3.304754] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.308624] vdc: detected capacity change from 0 to 2621440000 [ 3.324541] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.327489] vdd: detected capacity change from 0 to 2621440000 [ 3.340704] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.343062] vde: detected capacity change from 0 to 4294967296 [ 3.355933] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.358288] vdf: detected capacity change from 0 to 4294967296 [ 3.364890] libphy: Fixed MDIO Bus: probed [ 3.377769] usbcore: registered new interface driver usbserial_generic [ 3.380616] usbserial: USB Serial support registered for generic [ 3.383102] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.388257] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.390253] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.393095] mousedev: PS/2 mouse device common for all mice [ 3.396267] rtc_cmos 00:05: RTC can wake from S4 [ 3.400102] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.400172] rtc_cmos 00:05: registered as rtc0 [ 3.405536] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.407468] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.409947] intel_pstate: CPU model not supported [ 3.416368] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.420743] hid: raw HID events driver (C) Jiri Kosina [ 3.423205] usbcore: registered new interface driver usbhid [ 3.425479] usbhid: USB HID core driver [ 3.426892] drop_monitor: Initializing network drop monitor service [ 3.428740] Initializing XFRM netlink socket [ 3.430901] NET: Registered protocol family 10 [ 3.434300] Segment Routing with IPv6 [ 3.435977] NET: Registered protocol family 17 [ 3.438221] mpls_gso: MPLS GSO support [ 3.443571] RAS: Correctable Errors collector initialized. [ 3.445626] AVX version of gcm_enc/dec engaged. [ 3.447220] AES CTR mode by8 optimization enabled [ 3.522705] sched_clock: Marking stable (3522683469, 0)->(4412196592, -889513123) [ 3.525758] registered taskstats version 1 [ 3.527789] Loading compiled-in X.509 certificates [ 3.531206] zswap: loaded using pool lzo/zbud [ 3.560296] Key type big_key registered [ 3.580408] Key type encrypted registered [ 3.582776] ima: No TPM chip found, activating TPM-bypass! [ 3.586258] ima: Allocated hash algorithm: sha1 [ 3.588052] ima: No architecture policies found [ 3.590284] evm: Initialising EVM extended attributes: [ 3.592877] evm: security.selinux [ 3.594432] evm: security.ima [ 3.595982] evm: security.capability [ 3.597578] evm: HMAC attrs: 0x1 [ 3.600496] rtc_cmos 00:05: setting system clock to 2026-09-05 01:59:50 UTC (1788573590) [ 3.607428] debug: unmapping init [mem 0xffffffffb4403000-0xffffffffb45fffff] [ 3.611087] debug: unmapping init [mem 0xffffffffb3182000-0xffffffffb3458fff] [ 3.620412] Write protecting the kernel read-only data: 28672k [ 3.624083] debug: unmapping init [mem 0xffffffffb1803000-0xffffffffb19fffff] [ 3.627349] debug: unmapping init [mem 0xffffffffb2114000-0xffffffffb21fffff] [ 3.670134] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.679900] systemd[1]: Detected virtualization kvm. [ 3.682149] systemd[1]: Detected architecture x86-64. [ 3.683775] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.708832] systemd[1]: No hostname configured. [ 3.710454] systemd[1]: Set hostname to . [ 3.712983] random: systemd: uninitialized urandom read (16 bytes read) [ 3.715650] systemd[1]: Initializing machine ID from random generator. [ 3.760594] random: ln: uninitialized urandom read (6 bytes read) [ 3.841541] random: systemd: uninitialized urandom read (16 bytes read) [ 3.845399] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.850982] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.859793] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.536445] device-mapper: uevent: version 1.0.3 [ 4.538731] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.334917] virtio_net virtio0 ens2: renamed from eth0 [ 5.337422] random: fast init done Starting dracut initqueue hook... [ 5.407454] scsi host0: ata_piix [ 5.434310] scsi host1: ata_piix [ 5.436209] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.438241] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.157438] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.103748] random: crng init done [ 10.105473] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.646834] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.767105] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.029918] SELinux: Disabled at runtime. [ 12.089903] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.099890] systemd[1]: Detected virtualization kvm. [ 12.102011] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.621583] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.626412] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.632102] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.636508] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.639828] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.647797] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.660311] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ 12.808930] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.106506] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.343212] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.426199] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.560205] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.577703] EDAC sbridge: Ver: 1.1.2 [ 15.434568] Key type dns_resolver registered [ 15.745512] NFS: Registering the id_resolver key type [ 15.747762] Key type id_resolver registered [ 15.749471] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg329-server login: [ 42.848292] libcfs: loading out-of-tree module taints kernel. [ 42.872212] Key type ._llcrypt registered [ 42.874313] Key type .llcrypt registered [ 42.928568] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 60.876990] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 61.310997] hrtimer: interrupt took 5988906 ns [ 62.828380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.847415] alg: No test for adler32 (adler32-zlib) [ 64.592062] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 65.677022] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 67.520172] Key type lgssc registered [ 69.411220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 91.722507] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 140.073627] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 153.735322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 153.809650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 155.059620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 155.100790] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 155.217361] Lustre: lustre-MDT0000: new disk, initializing [ 155.288560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 155.315164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 159.434422] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.040155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 173.147813] Lustre: 6525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 173.192565] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 173.198705] Lustre: Skipped 1 previous similar message [ 173.291555] Lustre: lustre-MDT0001: new disk, initializing [ 173.360624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 173.384705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 173.400127] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 177.570631] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 182.700684] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 192.882996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 193.145544] Lustre: lustre-OST0000: new disk, initializing [ 193.153181] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 193.158675] Lustre: 8462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 193.225627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.267252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 194.276683] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 194.371219] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 199.469371] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.098765] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 213.297595] Lustre: lustre-OST0001: new disk, initializing [ 213.301746] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 213.308267] Lustre: 9535:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 213.397411] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 219.640668] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.653213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 219.660590] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 219.707843] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 232.504921] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 238.782192] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 244.974632] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing check_logdir /tmp/testlogs/ [ 250.234850] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing yml_node [ 255.510513] Lustre: DEBUG MARKER: Client: 2.17.58.4 [ 258.834806] Lustre: DEBUG MARKER: MDS: 2.17.58.4 [ 261.463726] Lustre: DEBUG MARKER: OSS: 2.17.58.4 [ 263.456784] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 4 22:04:07 EDT 2026 [ 281.417607] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 282.802918] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 301.537332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.538450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.554301] Lustre: Skipped 1 previous similar message [ 301.565470] Lustre: Skipped 3 previous similar messages [ 305.020708] Lustre: server umount lustre-MDT0000 complete [ 308.816766] LustreError: 6517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573895 with bad export cookie 16252613308606877851 [ 308.820706] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 308.829131] LustreError: 6517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 311.787061] LustreError: 6529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 311.802405] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 311.811370] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 311.831034] LustreError: 6529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 311.880578] Lustre: Skipped 3 previous similar messages [ 313.824991] LustreError: 6531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 313.826713] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 313.838639] LustreError: 6531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 313.852602] Lustre: Skipped 2 previous similar messages [ 315.090926] Lustre: server umount lustre-MDT0001 complete [ 318.511647] Lustre: server umount lustre-OST0000 complete [ 322.074294] Lustre: server umount lustre-OST0001 complete [ 329.894340] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 341.850234] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 379.356880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 379.594321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 379.613290] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 379.684458] Lustre: lustre-MDT0000: new disk, initializing [ 379.793345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 379.805519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 384.449728] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 395.377924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 395.599247] Lustre: 15102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 395.699316] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 395.709558] Lustre: Skipped 1 previous similar message [ 395.813548] Lustre: lustre-MDT0001: new disk, initializing [ 395.955988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 396.000646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 396.010909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 400.384376] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 410.056861] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 416.581868] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 424.382108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 424.819901] Lustre: lustre-OST0000: new disk, initializing [ 424.827951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 424.834022] Lustre: 17071:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 424.923333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 426.651760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 426.669553] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 426.823589] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 431.537235] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 441.118491] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 446.222658] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 446.427554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 451.421813] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 451.715432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 457.699756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 457.700646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 457.716529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 457.740827] Lustre: Skipped 1 previous similar message [ 459.927782] Lustre: server umount lustre-OST0000 complete [ 467.944538] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.953170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 467.956934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 467.962852] Lustre: Skipped 1 previous similar message [ 467.991790] Lustre: Skipped 2 previous similar messages [ 470.286391] Lustre: server umount lustre-MDT0000 complete [ 473.061865] LustreError: 15115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 473.088548] LustreError: 15115:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 474.210247] LustreError: 15095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574061 with bad export cookie 16252613308606879384 [ 474.211680] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 474.226048] LustreError: 15095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 474.577402] Lustre: server umount lustre-MDT0001 complete [ 479.809308] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 22:07:44 (1788574064) [ 498.487867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 499.033375] LustreError: 19948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.157393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 503.900816] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 504.303186] LustreError: 19949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 512.712093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.381474] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 526.690255] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 533.128661] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 535.552684] Lustre: Failing over lustre-MDT0000 [ 535.805894] Lustre: server umount lustre-MDT0000 complete [ 538.592910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 538.605791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 538.638446] LustreError: 19948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.667084] LustreError: 19948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 554.977651] Lustre: 3644:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788574125/real 1788574125] req@ffff9684ecf62a00 x1875455405878272/t0(0) o400->MGC192.168.203.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788574141 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 555.024490] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 555.053270] LustreError: 19949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 555.078238] LustreError: 19949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 555.356237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.264844] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe18ce1b1b34c9c1e [ 564.271667] Lustre: MGC192.168.203.129@tcp: Connection restored to 0@lo (at 0@lo) [ 564.526026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 564.534622] Lustre: Skipped 1 previous similar message [ 569.248356] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 569.829044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 569.829548] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 569.861164] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 580.528604] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 585.186865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 585.199067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 585.221711] Lustre: Skipped 1 previous similar message [ 585.226433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 591.035865] Lustre: server umount lustre-MDT0000 complete [ 593.378039] LustreError: 19944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 593.403802] LustreError: 19944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 594.973978] LustreError: 19928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574181 with bad export cookie 16252613308606880798 [ 594.987840] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 594.999343] LustreError: 19928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 595.467109] Lustre: server umount lustre-MDT0001 complete [ 604.218159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 609.519304] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 618.537067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 625.089163] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 634.513104] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 642.642197] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 645.211266] Lustre: Failing over lustre-MDT0000 [ 645.522613] Lustre: server umount lustre-MDT0000 complete [ 648.678733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 648.698057] Lustre: Skipped 1 previous similar message [ 658.915137] LustreError: 23290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 658.929924] LustreError: 23290:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 665.063302] Lustre: 3643:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788574235/real 1788574235] req@ffff9684ecf61c00 x1875455405909888/t0(0) o400->MGC192.168.203.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788574251 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 665.103221] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 665.647578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 675.594519] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 680.862301] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 680.934130] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 680.941778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 680.951409] Lustre: Skipped 1 previous similar message [ 680.969875] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 690.260418] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 696.289492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 696.296318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 696.308232] Lustre: Skipped 1 previous similar message [ 696.316158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 696.319195] Lustre: Skipped 2 previous similar messages [ 700.612739] Lustre: server umount lustre-MDT0000 complete [ 704.382737] LustreError: 23270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574291 with bad export cookie 16252613308606881736 [ 704.390030] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 704.396360] LustreError: 23270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 704.734857] Lustre: server umount lustre-MDT0001 complete [ 712.958982] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 22:11:37 (1788574297) [ 729.439530] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 740.284897] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 782.259338] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 795.111864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 795.503662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 795.549313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 795.610867] Lustre: lustre-MDT0000: new disk, initializing [ 795.662302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 795.667877] Lustre: Skipped 3 previous similar messages [ 795.679202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 800.733636] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 811.646888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 811.808657] Lustre: 29669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 812.118283] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 816.150222] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 826.225295] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 832.833566] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 840.245077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 840.646822] Lustre: 31635:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 842.719128] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 842.759993] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 842.803522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 842.826576] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 848.495078] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 858.640531] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 864.801271] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 888.806690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 888.808521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 888.809475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 888.809480] Lustre: Skipped 2 previous similar messages [ 888.823180] Lustre: Skipped 2 previous similar messages [ 888.847546] LustreError: Skipped 1 previous similar message [ 891.415800] Lustre: server umount lustre-OST0000 complete [ 910.305058] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 910.489725] Lustre: server umount lustre-MDT0000 complete [ 913.383860] LustreError: 29679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 913.399316] LustreError: 29679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 914.417703] LustreError: 30472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574501 with bad export cookie 16252613308606882142 [ 914.429852] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 915.073739] Lustre: server umount lustre-MDT0001 complete [ 926.797893] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 929.587375] Key type lgssc unregistered [ 929.840964] LNet: 33611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.851769] LNetError: 33611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.877619] LNet: Removed LNI 192.168.203.129@tcp [ 930.771142] Key type .llcrypt unregistered [ 930.775766] Key type ._llcrypt unregistered [ 956.784394] Key type ._llcrypt registered [ 956.789565] Key type .llcrypt registered [ 956.913989] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 957.933198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 957.953874] alg: No test for adler32 (adler32-zlib) [ 959.115205] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 959.322084] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 961.016193] Key type lgssc registered [ 962.053591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 969.707334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 969.736716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 971.268214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 975.761547] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 981.475158] LustreError: 35021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 981.512820] LustreError: 35021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 986.306959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 986.593346] LustreError: 35022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 986.882225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 992.366050] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1002.320802] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1009.001529] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1017.022449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1017.443207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1023.815852] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1031.820359] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1032.315419] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1042.069598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1059.901706] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 22:17:23 (1788574643) [ 1063.392814] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1066.046681] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1075.571875] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 22:17:39 (1788574659) [ 1086.956869] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 22:17:50 (1788574670) [ 1088.261130] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1098.102076] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1098.110856] Lustre: Skipped 2 previous similar messages [ 1100.169523] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 22:18:04 (1788574684) [ 1102.664106] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1102.674581] Lustre: Skipped 2 previous similar messages [ 1111.866683] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1111.902965] Lustre: Skipped 4 previous similar messages [ 1129.701434] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 22:18:33 (1788574713) [ 1133.055954] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1137.418762] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1146.898550] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 22:18:51 (1788574731) [ 1148.704316] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1148.708581] Lustre: Skipped 6 previous similar messages [ 1162.523712] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 22:19:06 (1788574746) [ 1180.645314] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1180.646154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1180.675727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1182.762127] Lustre: server umount lustre-OST0000 complete [ 1190.881878] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1190.885425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1190.902843] Lustre: Skipped 2 previous similar messages [ 1190.913453] Lustre: Skipped 1 previous similar message [ 1193.778061] Lustre: server umount lustre-MDT0000 complete [ 1196.001184] LustreError: 37474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1196.021770] LustreError: 37474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1197.407245] LustreError: 39459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574784 with bad export cookie 12699841524121005624 [ 1197.413447] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1197.420270] LustreError: 39459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1197.729402] Lustre: server umount lustre-MDT0001 complete [ 1209.234656] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 1212.154679] Key type lgssc unregistered [ 1212.485426] LNet: 42908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1212.490747] LNetError: 42908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1213.550259] LNet: Removed LNI 192.168.203.129@tcp [ 1214.720218] Key type .llcrypt unregistered [ 1214.723835] Key type ._llcrypt unregistered [ 1233.923728] Key type ._llcrypt registered [ 1233.927806] Key type .llcrypt registered [ 1234.050277] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 1249.100970] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 1250.135415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1250.175440] alg: No test for adler32 (adler32-zlib) [ 1251.377609] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 1251.625857] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 1253.312161] Key type lgssc registered [ 1254.632481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1289.539687] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1289.583289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1290.987590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1291.022752] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1291.134075] Lustre: lustre-MDT0000: new disk, initializing [ 1291.216930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1291.245427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1296.551820] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1308.590707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1308.728515] Lustre: 46059:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1308.786731] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1308.790829] Lustre: Skipped 1 previous similar message [ 1308.856484] Lustre: lustre-MDT0001: new disk, initializing [ 1308.952089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1308.982508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1309.001123] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1314.677408] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1325.348989] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1332.095972] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1340.913944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1341.360237] Lustre: lustre-OST0000: new disk, initializing [ 1341.365748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1341.376699] Lustre: 48028:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1341.480442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.759119] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1350.822969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1350.849742] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1351.055459] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1362.467060] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1368.244395] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1368.598496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1374.301256] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1374.537766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1376.739428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1376.740796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1376.792307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1381.862222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1381.873415] Lustre: Skipped 2 previous similar messages [ 1382.759252] Lustre: server umount lustre-OST0000 complete [ 1386.979122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1387.004040] Lustre: Skipped 1 previous similar message [ 1387.022332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1387.029819] Lustre: Skipped 1 previous similar message [ 1392.114772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1392.120623] Lustre: Skipped 1 previous similar message [ 1397.226178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1397.237796] Lustre: Skipped 1 previous similar message [ 1401.315588] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1401.470994] Lustre: server umount lustre-MDT0000 complete [ 1405.432938] LustreError: 48027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574992 with bad export cookie 833021652023248581 [ 1405.437509] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1405.444928] LustreError: 48027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1405.796956] Lustre: server umount lustre-MDT0001 complete [ 1416.702162] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 22:23:20 (1788575000) [ 1425.211499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1425.873980] LustreError: 50491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1425.912947] LustreError: 50491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1425.972602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1431.012858] LustreError: 50492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1431.712801] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1435.108868] LustreError: 50491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1440.229299] LustreError: 50492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1442.949983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1443.335052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1448.403603] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1456.235955] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1462.503270] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1470.105502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1470.537515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1478.356857] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1486.774908] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1494.587647] LustreError: 53021:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1495.615682] Lustre: Modifying parameter general.timeout=1 in log params [ 1497.463825] Lustre: Modifying parameter general.timeout=3 in log params [ 1497.468130] Lustre: Skipped 1 previous similar message [ 1499.485507] Lustre: Modifying parameter general.timeout=5 in log params [ 1499.497470] Lustre: Skipped 1 previous similar message [ 1504.152693] Lustre: Modifying parameter general.timeout=10 in log params [ 1504.162459] Lustre: Skipped 4 previous similar messages [ 1512.809326] Lustre: Modifying parameter general.timeout=19 in log params [ 1512.823024] Lustre: Skipped 8 previous similar messages [ 1529.517336] Lustre: Modifying parameter general.timeout=37 in log params [ 1529.528178] Lustre: Skipped 17 previous similar messages [ 1561.989490] Lustre: Modifying parameter general.timeout=71 in log params [ 1561.993801] Lustre: Skipped 33 previous similar messages [ 1625.071084] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 22:26:49 (1788575209) [ 1626.136890] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1626.145208] Lustre: Skipped 59 previous similar messages [ 1627.941323] Lustre: 59976:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1629.875205] Lustre: 60074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1631.893766] Lustre: 60172:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1637.604368] Lustre: 60483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1637.615118] Lustre: 60483:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1667.913550] Lustre: 61841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1667.942585] LustreError: 61841:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1673.287618] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 22:27:36 (1788575256) [ 1688.035114] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1688.036992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1688.049194] Lustre: Skipped 3 previous similar messages [ 1688.053459] Lustre: Skipped 1 previous similar message [ 1691.626267] Lustre: server umount lustre-MDT0000 complete [ 1693.153664] LustreError: 50486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1693.178594] LustreError: 50486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1695.317100] LustreError: 61164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575282 with bad export cookie 833021652023249456 [ 1695.332610] LustreError: 61164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1695.333575] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1695.726395] Lustre: server umount lustre-MDT0001 complete [ 1709.306592] Lustre: server umount lustre-OST0000 complete [ 1735.428780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1735.664644] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1735.904120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1740.206832] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1749.297055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1749.490725] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1749.573995] Lustre: 64371:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1753.997669] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1761.403662] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1766.813042] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1773.067165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1773.311748] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1773.700572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1773.713755] Lustre: Skipped 1 previous similar message [ 1780.255196] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1787.454203] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1788.898761] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1788.905347] Lustre: Skipped 22 previous similar messages [ 1788.942134] Lustre: 66939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1797.601690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1797.608348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1797.618712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1809.888292] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1810.125987] Lustre: server umount lustre-OST0000 complete [ 1816.550485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1816.554266] LustreError: Skipped 1 previous similar message [ 1816.564263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1816.583023] Lustre: Skipped 1 previous similar message [ 1828.320196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1828.549746] Lustre: server umount lustre-MDT0000 complete [ 1828.833893] LustreError: 64379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1828.852441] LustreError: 64379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1832.173443] LustreError: 64364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575419 with bad export cookie 833021652023252557 [ 1832.176375] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1832.184147] LustreError: 64364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1832.589469] Lustre: server umount lustre-MDT0001 complete [ 1843.538281] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 1846.859955] Key type lgssc unregistered [ 1847.115983] LNet: 68152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1847.122844] LNetError: 68152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1847.137731] LNet: Removed LNI 192.168.203.129@tcp [ 1847.830840] Key type .llcrypt unregistered [ 1847.836789] Key type ._llcrypt unregistered [ 1873.815642] Key type ._llcrypt registered [ 1873.825046] Key type .llcrypt registered [ 1874.020606] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 1875.310930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1875.372807] alg: No test for adler32 (adler32-zlib) [ 1876.468440] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 1876.765709] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 1878.448681] Key type lgssc registered [ 1879.690896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1889.410259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1889.450060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1891.011564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1897.024489] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1900.521124] LustreError: 69562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1900.537557] LustreError: 69562:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1905.633487] LustreError: 69563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1906.167857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1906.829688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1912.129125] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1919.973750] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1925.246765] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1931.460170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1931.938900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1936.641511] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1943.598717] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1951.886466] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1961.955823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1961.960564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1961.976790] Lustre: Skipped 2 previous similar messages [ 1962.009468] Lustre: Skipped 2 previous similar messages [ 1966.304963] Lustre: server umount lustre-MDT0000 complete [ 1967.074175] LustreError: 69563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1967.099166] LustreError: 69563:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1970.209589] LustreError: 70305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575557 with bad export cookie 6319942195465408715 [ 1970.211946] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1970.236483] LustreError: 70305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1970.565321] Lustre: server umount lustre-MDT0001 complete [ 1984.492381] Lustre: server umount lustre-OST0000 complete [ 2016.950975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2017.217259] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2017.255280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2017.531858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2021.703945] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2030.294939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2030.499565] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2030.561669] Lustre: 74438:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2030.724783] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2031.288988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2037.061907] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2044.448196] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2050.382534] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2057.149760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2057.332766] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2057.775027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2064.033943] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2071.058856] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2072.553657] Lustre: 77013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2074.279862] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2083.297414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2083.301323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2083.315488] LustreError: Skipped 1 previous similar message [ 2083.332488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2083.336806] Lustre: Skipped 1 previous similar message [ 2084.699331] Lustre: server umount lustre-OST0000 complete [ 2088.427295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2088.432394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2088.440777] Lustre: Skipped 2 previous similar messages [ 2088.451573] Lustre: Skipped 1 previous similar message [ 2093.545153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2093.552201] Lustre: Skipped 1 previous similar message [ 2094.243449] Lustre: server umount lustre-MDT0000 complete [ 2097.521948] LustreError: 75183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575684 with bad export cookie 6319942195465409562 [ 2097.532355] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2097.533737] LustreError: 75183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2097.807324] Lustre: server umount lustre-MDT0001 complete [ 2107.589316] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 2109.629569] Key type lgssc unregistered [ 2109.979489] LNet: 78222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2109.983567] LNetError: 78222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2111.030343] LNet: Removed LNI 192.168.203.129@tcp [ 2111.841089] Key type .llcrypt unregistered [ 2111.843426] Key type ._llcrypt unregistered [ 2132.997390] Key type ._llcrypt registered [ 2133.001548] Key type .llcrypt registered [ 2133.121498] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 2134.065587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2134.123397] alg: No test for adler32 (adler32-zlib) [ 2135.281756] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 2135.505225] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 2137.184201] Key type lgssc registered [ 2138.504989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2145.217419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2145.259672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2146.811391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2151.909378] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2157.029504] LustreError: 79630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2157.053996] LustreError: 79630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2161.869599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2162.518788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2167.170680] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2174.542253] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2180.763729] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2188.182664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2188.689329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2194.971126] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2202.363963] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2212.192794] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2213.280703] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2215.123082] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2215.132698] Lustre: Skipped 1 previous similar message [ 2217.842578] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2217.847645] Lustre: Skipped 2 previous similar messages [ 2222.075044] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2222.083393] Lustre: Skipped 4 previous similar messages [ 2230.634602] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2230.640924] Lustre: Skipped 8 previous similar messages [ 2247.413098] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2247.422306] Lustre: Skipped 18 previous similar messages [ 2279.902244] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2279.905964] Lustre: Skipped 39 previous similar messages [ 2302.953604] Lustre: 87724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2306.008548] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 22:38:10 (1788575890) [ 2307.918875] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2310.508369] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 22:38:14 (1788575894) [ 2315.748502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2315.750827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2315.755730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2320.874821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2320.881241] Lustre: Skipped 2 previous similar messages [ 2321.388328] Lustre: server umount lustre-OST0000 complete [ 2325.986447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2325.988379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2325.991307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2326.002669] Lustre: Skipped 2 previous similar messages [ 2326.032562] Lustre: Skipped 1 previous similar message [ 2330.827520] Lustre: server umount lustre-MDT0000 complete [ 2331.105201] LustreError: 79629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2331.113733] LustreError: 79629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2333.568943] LustreError: 80369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575920 with bad export cookie 17174731974560617745 [ 2333.575069] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2333.583128] LustreError: 80369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2333.749947] Lustre: server umount lustre-MDT0001 complete [ 2342.151635] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 2344.288636] Key type lgssc unregistered [ 2344.536969] LNet: 88974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2344.545234] LNetError: 88974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2344.568645] LNet: Removed LNI 192.168.203.129@tcp [ 2345.216428] Key type .llcrypt unregistered [ 2345.220731] Key type ._llcrypt unregistered [ 2348.360985] Key type ._llcrypt registered [ 2348.364814] Key type .llcrypt registered [ 2348.441555] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2351.847888] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules [ 2352.422252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2352.442192] alg: No test for adler32 (adler32-zlib) [ 2353.532547] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 2353.706389] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 2355.376202] Key type lgssc registered [ 2356.220337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2357.276224] LustreError: 90171:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2372.520198] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 2379.437300] LustreError: 90171:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2381.164175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2381.193603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2382.647969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2386.115788] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2392.787937] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 22:39:37 (1788575977) [ 2393.062600] LustreError: 91194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2393.083662] LustreError: 91194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2396.615719] LustreError: 91226:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2396.623359] LustreError: 91226:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2396.628172] LustreError: 91226:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 2396.921737] Lustre: server umount lustre-MDT0000 complete [ 2407.232198] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 2409.505217] Key type lgssc unregistered [ 2409.703520] LNet: 92717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2409.707765] LNetError: 92717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2409.725170] LNet: Removed LNI 192.168.203.129@tcp [ 2410.350983] Key type .llcrypt unregistered [ 2410.352788] Key type ._llcrypt unregistered [ 2426.197575] Key type ._llcrypt registered [ 2426.200996] Key type .llcrypt registered [ 2426.318624] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 2427.272989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2427.291424] alg: No test for adler32 (adler32-zlib) [ 2428.320594] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 2428.509574] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 2430.145476] Key type lgssc registered [ 2431.080360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2437.923137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2437.936339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2439.317462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2443.627298] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2449.891394] LustreError: 93929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2449.904871] LustreError: 93929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2450.886309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2451.441835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2455.157448] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2461.801139] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2468.167065] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2474.271201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2474.895871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2480.595218] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2487.707457] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2535.852092] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 22:41:59 (1788576119) [ 2537.323545] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2539.032155] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 22:42:03 (1788576123) [ 2543.589911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2543.604202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2543.619506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2545.633475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2545.636410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2545.644290] Lustre: Skipped 1 previous similar message [ 2545.668354] Lustre: Skipped 2 previous similar messages [ 2549.226613] Lustre: server umount lustre-MDT0000 complete [ 2550.761461] LustreError: 93929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2550.781863] LustreError: 93929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2552.480798] LustreError: 93909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576139 with bad export cookie 1770046742114898281 [ 2552.485696] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2552.492750] LustreError: 93909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2552.783421] Lustre: server umount lustre-MDT0001 complete [ 2566.643838] Lustre: server umount lustre-OST0000 complete [ 2577.580882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2577.946486] LustreError: 98204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2578.103363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2582.829794] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2583.524435] LustreError: 98205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2583.538155] LustreError: 98205:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2587.632354] LustreError: 98204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2591.024736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2591.732404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2596.056784] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2601.711683] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2606.850340] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2617.634541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2617.810315] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2617.815368] LustreError: 98191:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2617.838664] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2617.847332] LustreError: 100286:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2617.854615] LustreError: 100286:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2617.859970] LustreError: 100286:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2617.934461] Lustre: server umount lustre-OST0000 complete [ 2617.939940] LustreError: 100286:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2624.661096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2624.802405] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2624.807256] LustreError: 98194:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2624.812230] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2624.820307] LustreError: 100748:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2624.825113] LustreError: 100748:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2624.829132] LustreError: 100748:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2624.871404] Lustre: server umount lustre-OST0000 complete [ 2624.873282] LustreError: 100748:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2632.669839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2632.847273] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2632.851631] Lustre: Found index 0 for lustre-OST0000, updating log [ 2632.856218] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2632.909535] Lustre: lustre-OST0000: new disk, initializing [ 2632.914908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2632.921969] Lustre: 101282:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2633.189720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2639.199440] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2643.006128] Lustre: Failing over lustre-OST0000 [ 2643.148965] Lustre: server umount lustre-OST0000 complete [ 2646.498963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2646.499760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2646.508846] Lustre: Skipped 1 previous similar message [ 2651.620883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2651.629125] Lustre: Skipped 2 previous similar messages [ 2652.426104] Lustre: server umount lustre-MDT0000 complete [ 2655.407799] LustreError: 98186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576242 with bad export cookie 1770046742114901025 [ 2655.408649] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2655.417756] LustreError: 98186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2666.155599] Lustre: server umount lustre-MDT0001 complete [ 2674.763545] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 22:44:19 (1788576259) [ 2694.523086] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 2705.949545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2706.349529] LustreError: 103628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2706.470286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2710.956319] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2718.538757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2722.900499] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2725.447255] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2730.968428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2731.485234] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2732.521203] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2732.530476] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2732.544926] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2737.044731] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2737.641820] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2737.654613] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2742.766161] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 2742.777301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2743.425349] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2743.470573] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2743.525328] Lustre: lustre-OST0001: new disk, initializing [ 2743.528118] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2743.532923] Lustre: 105948:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2743.725985] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2743.730411] Lustre: Skipped 2 previous similar messages [ 2744.771880] Lustre: lustre-OST0000: Recovery over after 0:12, of 1 clients 1 recovered and 0 were evicted. [ 2744.771948] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2748.685905] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2752.572737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2752.585502] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2752.668984] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2755.542300] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2761.068601] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2766.911631] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.29@tcp (stopping) [ 2768.354300] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2768.378951] Lustre: Skipped 2 previous similar messages [ 2771.654598] Lustre: server umount lustre-MDT0001 complete [ 2771.991398] LustreError: 103623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2772.002239] LustreError: 103623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2780.598655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2780.693653] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2780.697993] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2780.770782] Lustre: 103617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2782.268184] LustreError: 107508:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2782.280258] LustreError: 107508:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2782.286685] LustreError: 103620:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2782.391789] LustreError: 107525:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.129@tcp: cfg command failed: rc = -17 [ 2782.399045] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2782.399045] [ 2786.073623] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2786.797263] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2795.117415] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 22:46:19 (1788576379) [ 2808.207246] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 2822.500157] Lustre: 109564:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2843.965989] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2846.906912] Lustre: 110708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2929.633108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2929.638944] LustreError: Skipped 2 previous similar messages [ 2929.651766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2929.665759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2929.668799] Lustre: Skipped 3 previous similar messages [ 2935.704212] Lustre: server umount lustre-MDT0000 complete [ 2939.292311] LustreError: 103608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576526 with bad export cookie 1770046742114901956 [ 2939.297455] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2939.303264] LustreError: 103608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2939.766594] Lustre: server umount lustre-MDT0001 complete [ 2954.123436] Lustre: server umount lustre-OST0000 complete [ 2956.260284] Lustre: 93239:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788576527/real 1788576527] req@ffff9684f6e11500 x1875457883845632/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788576543 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2956.283096] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2956.298980] Lustre: Skipped 3 previous similar messages [ 2957.696790] Lustre: server umount lustre-OST0001 complete [ 2963.028579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2971.500880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2979.169639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2988.158055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3007.135378] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 3017.083792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3017.100240] Lustre: lustre-MDT0000: reset Object Index mappings [ 3017.578970] LustreError: 114652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3017.597862] LustreError: 114652:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3017.703355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3017.710337] Lustre: Skipped 1 previous similar message [ 3021.733900] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3029.485802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3029.505904] Lustre: lustre-MDT0001: reset Object Index mappings [ 3034.636352] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3037.564056] Lustre: 115800:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3045.007174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3052.525991] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3055.114368] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 3060.694038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3062.395621] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3062.397095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3062.521134] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3067.191346] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3074.081675] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3077.470796] Lustre: 117679:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3126.353941] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 22:51:50 (1788576710) [ 3152.895965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3152.913407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3152.938743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3152.945956] Lustre: Skipped 8 previous similar messages [ 3159.297142] Lustre: server umount lustre-MDT0000 complete [ 3160.051722] LustreError: 114653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3160.060694] LustreError: 114653:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3162.697495] LustreError: 114634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576749 with bad export cookie 1770046742114948807 [ 3162.700028] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3162.706172] LustreError: 114634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3169.132731] Lustre: server umount lustre-MDT0001 complete [ 3178.614548] Lustre: server umount lustre-OST0000 complete [ 3196.384129] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3196.549059] Lustre: server umount lustre-OST0001 complete [ 3202.323541] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 3212.658483] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 3242.850737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3243.018732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3243.041994] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3243.125284] Lustre: lustre-MDT0000: new disk, initializing [ 3243.201948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3243.210240] Lustre: Skipped 3 previous similar messages [ 3243.226834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3247.194807] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3253.317269] Lustre: server umount lustre-MDT0000 complete [ 3261.338900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3261.589322] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3261.607346] Lustre: 121948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3261.617369] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3265.783756] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3269.907171] Lustre: server umount lustre-MDT0000 complete [ 3277.360860] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 22:54:21 (1788576861) [ 3278.992763] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3280.712135] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 22:54:24 (1788576864) [ 3297.629021] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 3309.199415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3314.180131] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3322.035393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3322.131430] Lustre: 124000:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3322.203317] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3322.207371] Lustre: Skipped 1 previous similar message [ 3322.295968] Lustre: lustre-MDT0001: new disk, initializing [ 3322.452881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3322.477143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3326.615361] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3331.334873] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3338.563408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3338.866865] Lustre: lustre-OST0000: new disk, initializing [ 3338.874200] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3338.888612] Lustre: 125578:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3340.461110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3340.496978] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3340.567072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3345.204294] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3355.247849] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3355.399368] Lustre: lustre-OST0001: new disk, initializing [ 3355.405188] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3355.411339] Lustre: 126453:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3356.666323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3356.683433] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3356.871691] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3361.937681] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3372.155236] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3376.757127] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3769.396424] Lustre: 132876:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3769.409726] Lustre: 132876:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3769.420120] Lustre: 132876:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3769.435737] Lustre: 132876:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3769.446588] Lustre: 132876:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3769.466539] Lustre: 132876:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4101.349227] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:08:04 (1788577684) [ 4115.941346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4115.947277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4115.965839] Lustre: Skipped 6 previous similar messages [ 4115.973328] Lustre: Skipped 8 previous similar messages [ 4121.068048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4121.074906] Lustre: Skipped 6 previous similar messages [ 4130.273534] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4130.755840] Lustre: server umount lustre-MDT0000 complete [ 4131.298643] LustreError: 124011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4131.329068] LustreError: 124011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4134.588762] LustreError: 123993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577721 with bad export cookie 1770046742115016616 [ 4134.599525] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4134.609683] LustreError: 123993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4136.429680] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4136.438233] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4136.448278] Lustre: Skipped 3 previous similar messages [ 4136.454692] Lustre: Skipped 4 previous similar messages [ 4141.360371] Lustre: server umount lustre-MDT0001 complete [ 4145.747338] Lustre: 127260:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (36/6s); client may timeout req@ffff9683cfa6ad80 x1875457892483456/t4294968369(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:91/0 lens 544/432 e 1 to 0 dl 1788577726 ref 1 fl Complete:/200/0 rc 0/0 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 4151.926458] Lustre: server umount lustre-OST0000 complete [ 4169.185656] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4169.441498] Lustre: server umount lustre-OST0001 complete [ 4177.351678] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 4188.193984] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 4227.667804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4228.029447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4228.081174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4228.227914] Lustre: lustre-MDT0000: new disk, initializing [ 4228.368807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4228.386098] Lustre: Skipped 5 previous similar messages [ 4228.489920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4233.387523] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4243.965029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4244.127912] Lustre: 135938:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4244.244238] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4244.257427] Lustre: Skipped 1 previous similar message [ 4244.369268] Lustre: lustre-MDT0001: new disk, initializing [ 4244.582397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4244.598272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4249.153228] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4258.584304] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4264.067787] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4272.389380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4272.819965] Lustre: lustre-OST0000: new disk, initializing [ 4272.823988] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4272.831200] Lustre: 137909:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4272.990920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4272.999093] Lustre: Skipped 1 previous similar message [ 4273.959208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4273.967788] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4274.132864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4280.348719] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4291.146138] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4300.839521] Lustre: lustre-MDD0000: changelog on [ 4303.831465] Lustre: lustre-MDD0001: changelog on [ 6121.590621] Lustre: lustre-MDD0001: changelog off [ 6123.431481] Lustre: 135946:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6124.491429] Lustre: lustre-MDD0000: changelog off [ 6132.712975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6132.714081] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6132.716529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6132.716536] Lustre: Skipped 3 previous similar messages [ 6132.725882] LustreError: Skipped 1 previous similar message [ 6132.777317] Lustre: Skipped 1 previous similar message [ 6137.612378] Lustre: server umount lustre-OST0000 complete [ 6142.946427] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6142.957417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6142.968666] Lustre: Skipped 2 previous similar messages [ 6147.417395] Lustre: server umount lustre-MDT0000 complete [ 6148.067032] LustreError: 137104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6148.085290] LustreError: 137104:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6150.507834] LustreError: 135933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579737 with bad export cookie 1770046742115843428 [ 6150.512955] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6150.519921] LustreError: 135933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6150.968260] Lustre: server umount lustre-MDT0001 complete [ 6161.364667] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 6163.887231] Key type lgssc unregistered [ 6164.250512] LNet: 147023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6164.272175] LNetError: 147023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6165.288936] LNet: Removed LNI 192.168.203.129@tcp [ 6166.293623] Key type .llcrypt unregistered [ 6166.302739] Key type ._llcrypt unregistered [ 6185.444026] Key type ._llcrypt registered [ 6185.446395] Key type .llcrypt registered [ 6185.538681] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 6200.404623] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 6201.566928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6201.658393] alg: No test for adler32 (adler32-zlib) [ 6202.795686] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 6203.104372] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 6204.793430] Key type lgssc registered [ 6205.997826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6237.777497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6237.812182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6239.107310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6239.145846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6239.219276] Lustre: lustre-MDT0000: new disk, initializing [ 6239.312580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6239.336046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6243.884075] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6253.919689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6254.099943] Lustre: 150219:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6254.190509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6254.204874] Lustre: Skipped 1 previous similar message [ 6254.243836] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6254.310642] Lustre: lustre-MDT0001: new disk, initializing [ 6254.446538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6254.494344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6254.518404] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6258.200955] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6265.612083] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6270.600876] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6276.483505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6276.713790] Lustre: lustre-OST0000: new disk, initializing [ 6276.719567] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6276.724924] Lustre: 152189:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6276.802055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6281.684614] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6285.888865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6285.900108] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6286.013348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6289.660408] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6295.280938] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_module obdecho/obdecho [ 6296.983570] LustreError: 153192:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6296.993187] LustreError: 153192:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6298.768274] LustreError: 153289:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6310.881687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6310.890671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6310.900655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6311.910749] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6311.911569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6311.934356] Lustre: Skipped 2 previous similar messages [ 6314.767485] Lustre: server umount lustre-MDT0000 complete [ 6317.989136] LustreError: 152188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579904 with bad export cookie 5948997363109893235 [ 6317.992468] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6317.999077] LustreError: 152188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6318.398661] Lustre: server umount lustre-MDT0001 complete [ 6332.563174] Lustre: server umount lustre-OST0000 complete [ 6351.830492] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 6361.606742] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 6395.534947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6395.726777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6395.749342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6395.840475] Lustre: lustre-MDT0000: new disk, initializing [ 6395.909703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6395.923804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6399.482778] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6408.153232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6408.257290] Lustre: 157004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6408.299741] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6408.303605] Lustre: Skipped 1 previous similar message [ 6408.403381] Lustre: lustre-MDT0001: new disk, initializing [ 6408.479875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6408.508255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6408.525493] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6412.406612] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6421.159194] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6426.035798] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6431.694314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6431.935289] Lustre: lustre-OST0000: new disk, initializing [ 6431.938378] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6431.944700] Lustre: 158972:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6432.043859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6437.677519] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6441.083604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6441.106456] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6441.238360] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6449.886657] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6455.662214] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6455.915631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6461.418928] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6461.668246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6467.047157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6467.054764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6467.064204] LustreError: Skipped 1 previous similar message [ 6467.076928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6467.078099] Lustre: Skipped 2 previous similar messages [ 6467.092409] Lustre: Skipped 1 previous similar message [ 6470.404258] Lustre: server umount lustre-OST0000 complete [ 6475.234259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6475.250124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6475.269035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6480.735947] Lustre: server umount lustre-MDT0000 complete [ 6482.409891] LustreError: 158064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6482.426126] LustreError: 158064:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6484.397813] LustreError: 157808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580071 with bad export cookie 5948997363109894187 [ 6484.407846] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6484.412921] LustreError: 157808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6484.873205] Lustre: server umount lustre-MDT0001 complete [ 6488.658246] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 23:47:52 (1788580072) [ 6490.266497] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6491.926890] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 23:47:56 (1788580076) [ 6498.851955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6499.266547] LustreError: 161385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6499.347806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6503.638069] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6504.423628] LustreError: 161386:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6509.541261] LustreError: 161385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6512.116921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6516.941383] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6524.470679] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6529.734473] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6535.646781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6535.976573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6535.981516] Lustre: Skipped 1 previous similar message [ 6542.032238] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6548.549495] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6553.569698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6553.579073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6553.596738] Lustre: Skipped 1 previous similar message [ 6553.603519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6553.612648] Lustre: Skipped 2 previous similar messages [ 6555.762976] Lustre: server umount lustre-MDT0000 complete [ 6558.707812] LustreError: 161367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580145 with bad export cookie 5948997363109895069 [ 6558.716723] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6558.718504] LustreError: 161367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6559.262296] LustreError: 161382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6568.868146] LustreError: 162674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6568.884601] LustreError: 162674:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6573.024189] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6573.237988] Lustre: server umount lustre-MDT0001 complete [ 6578.554404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6590.524171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6590.777075] LustreError: 165227:0:(llog.c:1655:llog_backup()) MGC192.168.203.129@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6590.782708] Lustre: 165227:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.129@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6598.641202] LustreError: 165232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6598.658783] LustreError: 165232:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6631.078396] LustreError: 163218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6631.090962] LustreError: 163218:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 28 previous similar messages [ 6651.360423] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6651.364818] Lustre: 165227:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.129@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6692.367975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6696.320498] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6697.442894] LustreError: 165252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6697.456982] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6697.461612] LustreError: 165252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 42 previous similar messages [ 6704.249301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6718.948905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6732.469832] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6733.809640] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6739.933250] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6741.761510] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6748.111460] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6749.891744] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6769.637188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6769.637956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6769.638538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6769.638544] Lustre: Skipped 10 previous similar messages [ 6769.656815] Lustre: Skipped 5 previous similar messages [ 6774.560666] Lustre: server umount lustre-MDT0000 complete [ 6778.529326] LustreError: 161365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580365 with bad export cookie 5948997363109901537 [ 6778.532479] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6778.552326] LustreError: 161365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6779.054253] Lustre: server umount lustre-MDT0001 complete [ 6792.430341] Lustre: server umount lustre-OST0000 complete [ 6812.727915] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 6822.135089] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 6856.006963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6856.284227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6856.371564] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6856.511699] Lustre: lustre-MDT0000: new disk, initializing [ 6856.640337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6856.649942] Lustre: Skipped 1 previous similar message [ 6856.667090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6861.081430] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6872.049414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6872.125841] Lustre: 170333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6872.400482] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6877.533182] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6887.803609] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6894.683915] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6904.805417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6905.119410] Lustre: lustre-OST0000: new disk, initializing [ 6905.122458] Lustre: Skipped 1 previous similar message [ 6905.125789] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6905.134577] Lustre: Skipped 2 previous similar messages [ 6905.145516] Lustre: 172304:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6912.069755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6912.075897] Lustre: Skipped 1 previous similar message [ 6912.080756] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6912.215957] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6912.239955] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6922.807352] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6927.993761] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6928.265545] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6933.174345] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6933.360778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6938.087844] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6938.096550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6938.101459] LustreError: Skipped 1 previous similar message [ 6938.105133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6938.117337] Lustre: Skipped 1 previous similar message [ 6938.126016] Lustre: Skipped 2 previous similar messages [ 6941.591760] Lustre: server umount lustre-OST0000 complete [ 6951.607785] Lustre: server umount lustre-MDT0000 complete [ 6953.444287] LustreError: 173497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6953.475204] LustreError: 173497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 31 previous similar messages [ 6954.991475] LustreError: 170326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580541 with bad export cookie 5948997363109918358 [ 6954.996064] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6955.003186] LustreError: 170326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6955.462807] Lustre: server umount lustre-MDT0001 complete [ 6964.171575] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 23:55:48 (1788580548) [ 6971.809698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6977.066494] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6985.849249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6991.570294] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6998.799298] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7004.537307] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7011.148097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7017.380519] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7025.613724] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7037.012192] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 23:57:00 (1788580620) [ 7043.557946] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7043.566911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7043.569954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7043.569962] Lustre: Skipped 2 previous similar messages [ 7043.577018] LustreError: Skipped 1 previous similar message [ 7043.631131] Lustre: Skipped 3 previous similar messages [ 7048.831945] Lustre: server umount lustre-OST0000 complete [ 7060.085449] Lustre: server umount lustre-MDT0000 complete [ 7064.535958] LustreError: 174695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580651 with bad export cookie 5948997363109919240 [ 7064.542401] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7064.549947] LustreError: 174695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7064.976198] Lustre: server umount lustre-MDT0001 complete [ 7079.580767] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 7083.190508] Key type lgssc unregistered [ 7083.687787] LNet: 178445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7083.695296] LNetError: 178445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7083.709793] LNet: Removed LNI 192.168.203.129@tcp [ 7084.876171] Key type .llcrypt unregistered [ 7084.884073] Key type ._llcrypt unregistered [ 7111.251175] Key type ._llcrypt registered [ 7111.253966] Key type .llcrypt registered [ 7111.360406] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 7112.326962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7112.448850] alg: No test for adler32 (adler32-zlib) [ 7114.238604] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 7114.535730] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 7116.211547] Key type lgssc registered [ 7117.785062] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7126.579408] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7126.601804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7142.304444] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7147.424547] LustreError: 179682:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.129@tcp: failed processing log, type 4: rc = -110 [ 7152.545372] LustreError: 179682:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7152.580724] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7152.589421] LustreError: MGC192.168.203.129@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7152.598338] LustreError: 179682:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7152.608239] LustreError: 179682:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 7152.614809] LustreError: 179682:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7152.785510] Lustre: server umount lustre-OST0000 complete [ 7152.792294] LustreError: 179682:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7163.679233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7165.380061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7170.693459] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7175.657790] LustreError: 180184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7175.703732] LustreError: 180184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7179.971872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7180.404919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7185.466624] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7194.602700] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7202.162776] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7211.541125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7211.720528] LustreError: 182000:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7211.746319] Lustre: 182000:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.129@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7212.027668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7220.374703] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7228.390029] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7231.459184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7231.484783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7231.510573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7236.067413] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7236.069828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7241.187245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7241.196985] Lustre: Skipped 4 previous similar messages [ 7245.280680] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7245.427772] Lustre: server umount lustre-MDT0000 complete [ 7246.308355] LustreError: 180178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7246.339449] LustreError: 180178:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7249.433073] LustreError: 180164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580836 with bad export cookie 10429241745508442171 [ 7249.442455] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7249.445794] LustreError: 180164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7249.707961] Lustre: server umount lustre-MDT0001 complete [ 7264.110332] Lustre: server umount lustre-OST0000 complete [ 7275.637303] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 7291.314788] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 7328.208152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7328.390669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7328.409751] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7328.479726] Lustre: lustre-MDT0000: new disk, initializing [ 7328.556331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7328.584379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7333.033922] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7343.146370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7343.302865] Lustre: 185563:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7343.356985] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7343.361878] Lustre: Skipped 1 previous similar message [ 7343.465785] Lustre: lustre-MDT0001: new disk, initializing [ 7343.584905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7343.636875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7343.642975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7348.441127] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7357.723677] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7363.567293] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7370.645643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7371.048422] Lustre: lustre-OST0000: new disk, initializing [ 7371.050488] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7371.061947] Lustre: 187532:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7371.139453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7378.195324] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7380.097604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7380.116627] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7380.340655] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7389.259520] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7393.560991] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7393.764828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7398.738520] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7399.027199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7400.932630] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7400.952176] Lustre: Skipped 2 previous similar messages [ 7400.952352] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7400.953239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7400.987958] LustreError: Skipped 1 previous similar message [ 7406.051267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7406.062095] Lustre: Skipped 1 previous similar message [ 7415.264182] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7415.414509] Lustre: server umount lustre-OST0000 complete [ 7420.392415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7420.398812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7420.424577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7420.437615] Lustre: Skipped 3 previous similar messages [ 7425.967173] Lustre: server umount lustre-MDT0000 complete [ 7426.530364] LustreError: 185571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7426.575790] LustreError: 185571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7430.608215] LustreError: 187543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581017 with bad export cookie 10429241745508442885 [ 7430.611530] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7430.622745] LustreError: 187543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7431.174479] Lustre: server umount lustre-MDT0001 complete [ 7442.359786] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 00:03:46 (1788581026) [ 7462.455474] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 7466.334820] Key type lgssc unregistered [ 7466.653858] LNet: 190480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7466.657954] LNetError: 190480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7466.680511] LNet: Removed LNI 192.168.203.129@tcp [ 7467.790770] Key type .llcrypt unregistered [ 7467.792742] Key type ._llcrypt unregistered [ 7485.068283] Key type ._llcrypt registered [ 7485.072379] Key type .llcrypt registered [ 7485.199203] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 7486.278128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7486.325167] alg: No test for adler32 (adler32-zlib) [ 7487.364979] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 7487.653193] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 7489.352164] Key type lgssc registered [ 7490.410949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7497.820178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7497.855909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7499.430053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7504.179861] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7509.990242] LustreError: 191696:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7510.007923] LustreError: 191696:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7511.868239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7512.201209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7516.362041] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7523.489185] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7529.847107] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7537.014151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7537.245322] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7537.250462] LustreError: 193559:0:(llog.c:1696:llog_backup()) MGC192.168.203.129@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7537.257450] Lustre: 193559:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.129@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7537.370666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7543.372624] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7550.468762] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7555.042492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7555.047332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7555.082483] Lustre: Skipped 2 previous similar messages [ 7555.113142] Lustre: Skipped 1 previous similar message [ 7559.412742] Lustre: server umount lustre-MDT0000 complete [ 7560.162644] LustreError: 191696:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7560.175160] LustreError: 191696:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7563.332145] LustreError: 191677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581150 with bad export cookie 13408993126118263392 [ 7563.335731] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7563.356115] LustreError: 191677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7563.860503] Lustre: server umount lustre-MDT0001 complete [ 7578.789871] Lustre: server umount lustre-OST0000 complete [ 7588.449792] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 7599.210835] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 7631.526524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7631.805229] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7631.837137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7631.910331] Lustre: lustre-MDT0000: new disk, initializing [ 7631.974824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7631.988807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7635.832249] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7645.541042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7645.655441] Lustre: 197082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7645.698827] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7645.702480] Lustre: Skipped 1 previous similar message [ 7645.706176] LustreError: 197866:0:(llog.c:1696:llog_backup()) MGC192.168.203.129@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7645.715760] LustreError: 197866:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7645.723131] Lustre: 197866:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.129@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7645.744420] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7645.750925] Lustre: Skipped 1 previous similar message [ 7645.845731] Lustre: lustre-MDT0001: new disk, initializing [ 7645.927823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7645.963239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7645.986088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7650.004830] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7659.226574] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7665.061835] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7671.566222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7671.845059] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7671.848437] Lustre: Skipped 1 previous similar message [ 7671.851429] LustreError: 199025:0:(llog.c:1696:llog_backup()) MGC192.168.203.129@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7671.859563] LustreError: 199025:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7671.867587] Lustre: 199025:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.129@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7671.880567] Lustre: 199025:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7671.909402] Lustre: lustre-OST0000: new disk, initializing [ 7671.922531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7671.944426] Lustre: 199052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7672.030378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7678.044490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7678.053084] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7678.164411] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7678.183062] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7687.635125] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7691.811273] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7692.049737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7696.523754] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7696.695190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7698.913696] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7698.925161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7698.953210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7698.972560] Lustre: Skipped 2 previous similar messages [ 7704.036290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7704.047186] Lustre: Skipped 1 previous similar message [ 7704.604836] Lustre: server umount lustre-OST0000 complete [ 7709.154781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7709.166437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7709.171591] Lustre: Skipped 2 previous similar messages [ 7709.186265] Lustre: Skipped 1 previous similar message [ 7714.313132] LustreError: 197096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7714.332575] LustreError: 197096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7714.384710] Lustre: server umount lustre-MDT0000 complete [ 7718.251619] LustreError: 197077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581305 with bad export cookie 13408993126118264113 [ 7718.267678] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7718.267960] LustreError: 197077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7718.598745] Lustre: server umount lustre-MDT0001 complete [ 7727.153848] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 00:08:31 (1788581311) [ 7744.120169] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 7754.192966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7754.584820] LustreError: 202197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7754.651044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7759.295572] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7759.852466] LustreError: 202198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7769.355449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7775.915753] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7779.455266] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7787.568824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7788.180614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7788.191284] Lustre: Skipped 1 previous similar message [ 7796.640303] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7805.581411] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7805.771193] Lustre: lustre-OST0001: new disk, initializing [ 7805.775494] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7805.781733] Lustre: 204502:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7812.772641] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7814.275064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7814.294688] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7814.341733] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7824.955534] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7830.023597] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7831.956961] Lustre: DEBUG MARKER: ADD OST3 [ 7836.669027] loop: module loaded [ 7836.744442] loop0: detected capacity change from 0 to 204800000 [ 7836.999938] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7837.023537] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7837.042807] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7843.574536] loop0: detected capacity change from 0 to 204800000 [ 7844.294572] Lustre: DEBUG MARKER: STOP OST3 [ 7849.679798] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7849.992542] Lustre: lustre-OST0003: new disk, initializing [ 7849.999635] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7850.012162] Lustre: 206514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7851.492119] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7851.666768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7851.681374] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7851.716229] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7853.537370] LustreError: 206668:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7853.638839] Lustre: Failing over lustre-OST0003 [ 7853.855948] Lustre: server umount lustre-OST0003 complete [ 7856.623639] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7856.627706] LustreError: 203681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7856.647025] Lustre: Skipped 1 previous similar message [ 7856.689580] LustreError: 203681:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7856.979553] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7865.006697] Lustre: DEBUG MARKER: START OST3 again [ 7872.047268] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7872.376889] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7872.383945] Lustre: Skipped 2 previous similar messages [ 7872.393615] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7872.548797] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7873.696230] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7873.699383] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7878.931274] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7899.361791] Lustre: Failing over lustre-OST0003 [ 7899.572841] Lustre: server umount lustre-OST0003 complete [ 7899.621488] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7899.625979] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7899.626838] LustreError: Skipped 2 previous similar messages [ 7899.636393] LustreError: 203683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7899.644538] Lustre: Skipped 1 previous similar message [ 7899.647380] LustreError: 203683:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7918.562801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7918.570138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7918.578641] Lustre: Skipped 1 previous similar message [ 7918.588093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7918.599121] Lustre: Skipped 3 previous similar messages [ 7923.698776] Lustre: server umount lustre-MDT0000 complete [ 7927.946553] LustreError: 205275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581514 with bad export cookie 13408993126118264995 [ 7927.955935] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7927.968289] LustreError: 205275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7928.804377] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7928.821319] Lustre: Skipped 1 previous similar message [ 7932.837258] LustreError: 203701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7932.894343] LustreError: 203701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [ 7934.722273] Lustre: server umount lustre-MDT0001 complete [ 7940.239961] Lustre: server umount lustre-OST0000 complete [ 7944.997422] Lustre: server umount lustre-OST0001 complete [ 7952.160315] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 7960.744766] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 7995.333031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7995.532340] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7995.552906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7995.622506] Lustre: lustre-MDT0000: new disk, initializing [ 7995.744535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8000.023965] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8009.923642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8010.031955] Lustre: 211964:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8010.260244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8010.266702] Lustre: Skipped 1 previous similar message [ 8010.307244] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8014.330513] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8023.369385] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8028.650405] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8034.706697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8034.961540] Lustre: lustre-OST0000: new disk, initializing [ 8034.964505] Lustre: Skipped 1 previous similar message [ 8034.968461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8034.978510] Lustre: Skipped 2 previous similar messages [ 8034.984277] Lustre: 213934:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8036.454672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8036.477400] Lustre: Skipped 1 previous similar message [ 8036.486163] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8036.579566] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8041.358821] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8050.589381] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8055.254435] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8055.437612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8059.570380] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8059.823235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8062.438489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8062.439857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8062.447873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8062.447883] Lustre: Skipped 2 previous similar messages [ 8062.448888] LustreError: Skipped 1 previous similar message [ 8062.504105] Lustre: Skipped 5 previous similar messages [ 8067.913337] Lustre: server umount lustre-OST0000 complete [ 8071.652688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8077.681273] Lustre: server umount lustre-MDT0000 complete [ 8077.799240] LustreError: 211973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8077.819254] LustreError: 211973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8081.601459] LustreError: 211957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581668 with bad export cookie 13408993126118267291 [ 8081.616211] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8081.619737] LustreError: 211957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8082.060607] Lustre: server umount lustre-MDT0001 complete [ 8085.671306] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 00:14:29 (1788581669) [ 8101.255465] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 8111.632675] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 8145.133888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8145.326629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8145.356570] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8145.443670] Lustre: lustre-MDT0000: new disk, initializing [ 8145.540514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8150.323538] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8161.084510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8161.201399] Lustre: 218541:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8161.685555] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8167.041062] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8177.944626] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8185.656765] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8193.016269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8193.305521] Lustre: 220511:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8195.129227] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8195.370167] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8200.871893] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8210.896068] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8217.204984] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8217.515780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8224.858949] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8225.273995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8231.398510] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8231.402794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8231.418301] LustreError: Skipped 1 previous similar message [ 8231.421708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8231.428325] Lustre: Skipped 3 previous similar messages [ 8231.455488] Lustre: Skipped 7 previous similar messages [ 8242.144451] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8242.423794] Lustre: server umount lustre-OST0000 complete [ 8252.810777] Lustre: server umount lustre-MDT0000 complete [ 8256.909744] LustreError: 218534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581843 with bad export cookie 13408993126118268166 [ 8256.940953] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8257.323776] Lustre: server umount lustre-MDT0001 complete [ 8266.647345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8267.135283] LustreError: 222732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8267.234933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8267.245724] Lustre: Skipped 4 previous similar messages [ 8271.882085] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8280.666220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8286.077519] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8294.968695] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8301.288472] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8308.653812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8316.285189] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8324.429615] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8335.841489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8335.855262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8335.862044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8335.875368] Lustre: Skipped 3 previous similar messages [ 8335.904488] Lustre: Skipped 9 previous similar messages [ 8340.920374] Lustre: server umount lustre-OST0000 complete [ 8351.693877] LustreError: 225610:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff9684d3c0ce00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 8351.812848] Lustre: server umount lustre-MDT0000 complete [ 8355.529373] LustreError: 222713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581942 with bad export cookie 13408993126118269055 [ 8355.542894] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8355.545404] LustreError: 222713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8355.843702] Lustre: server umount lustre-MDT0001 complete [ 8465.251417] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 8467.860803] Key type lgssc unregistered [ 8468.196676] LNet: 226399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8468.203773] LNetError: 226399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8468.224755] LNet: Removed LNI 192.168.203.129@tcp [ 8469.026176] Key type .llcrypt unregistered [ 8469.030473] Key type ._llcrypt unregistered [ 8491.952942] Key type ._llcrypt registered [ 8491.954996] Key type .llcrypt registered [ 8492.088880] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 8508.944434] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 8510.056064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8510.082820] alg: No test for adler32 (adler32-zlib) [ 8511.111817] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 8511.381672] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 8513.033070] Key type lgssc registered [ 8514.224426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8552.574763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8552.595359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8553.909991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8553.930762] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8554.033769] Lustre: lustre-MDT0000: new disk, initializing [ 8554.142125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8554.161280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8560.241833] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8571.679353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8571.750382] Lustre: 229748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8571.814836] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8571.818164] Lustre: Skipped 1 previous similar message [ 8571.892482] Lustre: lustre-MDT0001: new disk, initializing [ 8571.986561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8572.047076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8572.066182] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8578.071833] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8589.680966] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8596.172475] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8604.378056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8604.890311] Lustre: lustre-OST0000: new disk, initializing [ 8604.894615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8604.900544] Lustre: 231719:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8605.070699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8612.029876] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8613.962497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8613.975072] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8614.057938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8621.682140] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8626.704900] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8626.944873] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8631.526320] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8631.729820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8634.858033] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8634.863136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8634.866173] LustreError: Skipped 1 previous similar message [ 8634.868054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8634.891500] Lustre: Skipped 1 previous similar message [ 8639.881330] Lustre: server umount lustre-OST0000 complete [ 8645.094811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8645.123836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8645.142283] Lustre: Skipped 2 previous similar messages [ 8650.215498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8650.235185] Lustre: Skipped 1 previous similar message [ 8654.305650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8654.316251] Lustre: Skipped 1 previous similar message [ 8658.400251] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8658.746760] Lustre: server umount lustre-MDT0000 complete [ 8663.381061] LustreError: 229740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582250 with bad export cookie 13582951985506301767 [ 8663.395094] LustreError: 229740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8663.398325] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8664.038728] Lustre: server umount lustre-MDT0001 complete [ 8683.295974] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 8692.521955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8693.019580] LustreError: 234653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8693.055891] LustreError: 234653:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8693.094951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8697.199337] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8698.344691] LustreError: 234654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8702.437612] LustreError: 234653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8705.078522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8705.504922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8709.779121] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8712.433742] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8718.967508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8719.355710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8725.410272] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8735.002430] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8735.359445] Lustre: lustre-OST0001: new disk, initializing [ 8735.361683] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8735.375636] Lustre: 236961:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8735.512242] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8742.081358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8742.099120] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8742.218867] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8743.418408] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8755.023773] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8764.206368] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8777.708426] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 00:26:01 (1788582361) [ 8782.306678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8782.317901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8782.340044] Lustre: Skipped 1 previous similar message [ 8782.350373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8786.732155] Lustre: server umount lustre-MDT0000 complete [ 8788.457779] LustreError: 234654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8788.476343] LustreError: 234654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8790.170040] LustreError: 234635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582377 with bad export cookie 13582951985506302649 [ 8790.179130] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8790.184349] LustreError: 234635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8790.477151] Lustre: server umount lustre-MDT0001 complete [ 8805.027381] Lustre: server umount lustre-OST0000 complete [ 8818.600898] Lustre: server umount lustre-OST0001 complete [ 8824.599929] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 8833.725755] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 8868.640436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8868.840131] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8868.863854] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8868.993917] Lustre: lustre-MDT0000: new disk, initializing [ 8869.100045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8869.117521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8873.767881] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8883.745307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8883.855890] Lustre: 240903:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8883.987054] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8883.991968] Lustre: Skipped 1 previous similar message [ 8884.118802] Lustre: lustre-MDT0001: new disk, initializing [ 8884.311078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8884.331175] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8888.951807] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8898.275490] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8904.181485] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8911.502418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8911.769916] Lustre: lustre-OST0000: new disk, initializing [ 8911.774706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8911.782628] Lustre: 242874:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8913.301434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8913.310782] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8913.417982] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8917.838940] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8927.671373] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8932.300806] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8932.488188] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8936.465838] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8936.688034] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8938.979935] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8938.984690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8938.993656] LustreError: Skipped 1 previous similar message [ 8939.000571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8939.029763] Lustre: Skipped 4 previous similar messages [ 8939.046658] Lustre: Skipped 5 previous similar messages [ 8944.696262] Lustre: server umount lustre-OST0000 complete [ 8949.218371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8954.769409] Lustre: server umount lustre-MDT0000 complete [ 8958.584524] LustreError: 240894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582545 with bad export cookie 13582951985506304784 [ 8958.611472] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8958.615070] LustreError: 240894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8958.987966] Lustre: server umount lustre-MDT0001 complete [ 8969.021070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8969.812297] LustreError: 245090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8969.931886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8969.941712] Lustre: Skipped 2 previous similar messages [ 8976.078435] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8979.432151] LustreError: 245091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8979.471398] LustreError: 245091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8986.253864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8991.145151] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8998.039526] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9003.653863] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9010.630668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9017.815104] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9025.673127] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9172.961283] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9172.969094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9172.975950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9172.975958] Lustre: Skipped 6 previous similar messages [ 9173.009805] Lustre: Skipped 2 previous similar messages [ 9178.996601] Lustre: server umount lustre-OST0000 complete [ 9188.888664] Lustre: server umount lustre-MDT0000 complete [ 9191.869769] LustreError: 245071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582778 with bad export cookie 13582951985506305666 [ 9191.872307] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9191.882151] LustreError: 245071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9192.248916] Lustre: server umount lustre-MDT0001 complete [ 9292.527083] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 9295.007402] Key type lgssc unregistered [ 9295.307823] LNet: 248696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9295.322621] LNetError: 248696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9295.341888] LNet: Removed LNI 192.168.203.129@tcp [ 9296.251153] Key type .llcrypt unregistered [ 9296.255298] Key type ._llcrypt unregistered [ 9319.495266] Key type ._llcrypt registered [ 9319.497358] Key type .llcrypt registered [ 9319.580164] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 9333.353259] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 9334.354113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9334.467510] alg: No test for adler32 (adler32-zlib) [ 9335.557410] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [ 9335.753836] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 9337.368206] Key type lgssc registered [ 9338.385350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9377.557600] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 9387.115960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9387.146279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9388.371385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9388.418539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9388.511289] Lustre: lustre-MDT0000: new disk, initializing [ 9388.604264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9388.629267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9392.842487] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9402.872066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9402.987166] Lustre: 252766:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9403.066708] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9403.078100] Lustre: Skipped 1 previous similar message [ 9403.166572] Lustre: lustre-MDT0001: new disk, initializing [ 9403.324491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9403.399839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9403.408636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9407.037134] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9411.427562] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9417.876703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9418.175292] Lustre: lustre-OST0000: new disk, initializing [ 9418.184856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9418.196438] Lustre: 254397:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9418.286771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9419.996166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9420.009740] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9420.194196] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9424.462715] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9433.710829] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9433.863890] Lustre: lustre-OST0001: new disk, initializing [ 9433.871450] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9433.881727] Lustre: 255267:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9433.980811] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9439.569496] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9442.929523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9442.939525] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9443.054602] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9448.502055] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9454.405822] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9464.801663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9464.811532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9464.824986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9468.910554] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9468.913027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9468.931926] Lustre: Skipped 2 previous similar messages [ 9468.932951] Lustre: Skipped 2 previous similar messages [ 9470.323797] Lustre: server umount lustre-MDT0000 complete [ 9473.980294] LustreError: 254026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583060 with bad export cookie 4660910406162237086 [ 9473.986290] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9474.002183] LustreError: 254026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9474.023483] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9474.030315] LustreError: 250271:0:(import.c:761:ptlrpc_connect_import_locked()) already connecting [ 9474.033308] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9474.033315] Lustre: Skipped 1 previous similar message [ 9474.047219] Lustre: Skipped 1 previous similar message [ 9474.446614] Lustre: server umount lustre-MDT0001 complete [ 9480.506495] Lustre: server umount lustre-OST0000 complete [ 9484.130560] Lustre: server umount lustre-OST0001 complete [ 9497.115829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9506.964729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9516.559879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9525.796716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9542.838509] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 9551.908111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9551.920805] Lustre: lustre-MDT0000: reset Object Index mappings [ 9552.262227] LustreError: 260273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9552.278400] LustreError: 260273:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9552.343773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9556.387330] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9557.474419] LustreError: 260274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9562.612872] LustreError: 260273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9565.069994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9565.086931] Lustre: lustre-MDT0001: reset Object Index mappings [ 9565.477062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9569.819353] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9572.431957] Lustre: 261413:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9578.684900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9585.349405] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9590.328534] LustreError: 261766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9594.013643] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9594.450466] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9594.460310] Lustre: Skipped 1 previous similar message [ 9597.559596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9597.578214] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9601.224787] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9609.881460] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9613.992560] Lustre: 263287:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9621.988487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9622.016864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9622.062402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9622.071627] Lustre: Skipped 1 previous similar message [ 9627.077229] Lustre: server umount lustre-MDT0000 complete [ 9628.647596] LustreError: 260268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9628.669933] LustreError: 260268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9631.680860] LustreError: 260255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583218 with bad export cookie 4660910406162243876 [ 9631.682320] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9631.692683] LustreError: 260255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9632.348908] Lustre: server umount lustre-MDT0001 complete [ 9646.747410] Lustre: server umount lustre-OST0000 complete [ 9649.697042] Lustre: 250272:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788583220/real 1788583220] req@ffff9684c4187100 x1875465126143488/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788583236 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9649.730723] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9649.740493] Lustre: Skipped 3 previous similar messages [ 9650.626476] Lustre: server umount lustre-OST0001 complete [ 9667.634268] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 00:40:51 (1788583251) [ 9681.202649] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 9689.991392] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 9723.389528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9723.580562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9723.631845] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9723.738571] Lustre: lustre-MDT0000: new disk, initializing [ 9723.869713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9723.925770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9728.136633] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9737.267961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9737.358540] Lustre: 267541:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9737.405643] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9737.409558] Lustre: Skipped 1 previous similar message [ 9737.474919] Lustre: lustre-MDT0001: new disk, initializing [ 9737.574751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9737.586371] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9741.422751] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9750.121255] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9756.810591] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9763.786656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9764.079260] Lustre: lustre-OST0000: new disk, initializing [ 9764.089754] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9764.098612] Lustre: 269510:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9765.684079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9765.704151] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9765.821916] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9770.616112] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9779.138086] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9783.392274] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9783.667330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9788.519868] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9788.895073] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9791.457024] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9791.462636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9791.475613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9791.481976] Lustre: Skipped 1 previous similar message [ 9791.484145] Lustre: Skipped 4 previous similar messages [ 9797.445234] Lustre: server umount lustre-OST0000 complete [ 9801.708593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9801.719223] Lustre: Skipped 4 previous similar messages [ 9807.958648] Lustre: server umount lustre-MDT0000 complete [ 9811.940405] LustreError: 268433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9811.976579] LustreError: 268433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9812.360598] LustreError: 267532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583399 with bad export cookie 4660910406162245969 [ 9812.386938] LustreError: 267532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9812.387271] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9813.049516] Lustre: server umount lustre-MDT0001 complete [ 9832.910502] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 9842.745418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9843.081610] LustreError: 272440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9843.151167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9843.159471] Lustre: Skipped 2 previous similar messages [ 9847.107391] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9854.449576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9858.020626] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9860.405995] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9865.766987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9871.796609] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9879.979627] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9880.173529] Lustre: lustre-OST0001: new disk, initializing [ 9880.177987] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9880.184364] Lustre: 274746:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9881.624093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9881.638472] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9881.778942] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9885.895515] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9894.633284] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9897.586322] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9902.468916] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9902.498201] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9912.805257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9912.811277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9912.825094] Lustre: Skipped 5 previous similar messages [ 9912.840254] Lustre: Skipped 5 previous similar messages [ 9913.843947] Lustre: server umount lustre-MDT0000 complete [ 9917.154631] LustreError: 272422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583504 with bad export cookie 4660910406162246851 [ 9917.160843] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9917.173846] LustreError: 272422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9917.458477] Lustre: server umount lustre-MDT0001 complete [ 9930.327393] Lustre: server umount lustre-OST0000 complete [ 9933.281163] Lustre: 250271:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788583504/real 1788583504] req@ffff9683c46bce00 x1875465126243840/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788583520 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9934.084177] Lustre: server umount lustre-OST0001 complete [ 9949.326490] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 9956.775923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9957.046567] LustreError: 278091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9957.064242] LustreError: 278091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9960.580386] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9967.389265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9971.715997] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9974.110502] Lustre: 279230:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9979.709688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9980.041808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9980.046241] Lustre: Skipped 5 previous similar messages [ 9982.160416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9985.007352] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9992.069911] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9993.358928] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9997.731744] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10003.961764] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10007.238187] Lustre: 281109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10011.583589] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10011.613116] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10023.183167] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 00:46:47 (1788583607) [10039.784960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10039.789094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10039.791842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10039.801306] Lustre: Skipped 2 previous similar messages [10039.857665] LustreError: Skipped 1 previous similar message [10044.298931] Lustre: server umount lustre-MDT0000 complete [10044.901858] LustreError: 279414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10044.916991] LustreError: 279414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10047.514038] LustreError: 278071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583634 with bad export cookie 4660910406162253270 [10047.519983] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10047.525977] LustreError: 278071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10053.923965] Lustre: server umount lustre-MDT0001 complete [10063.447848] Lustre: server umount lustre-OST0000 complete [10081.248200] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10081.489129] Lustre: server umount lustre-OST0001 complete [10088.386110] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [10097.939617] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [10137.915121] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [10146.090549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10146.267468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10146.293772] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10146.356617] Lustre: lustre-MDT0000: new disk, initializing [10146.450041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10150.487517] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10160.205900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10160.353783] Lustre: 285636:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10160.620578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10164.511668] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10174.687572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10175.009441] Lustre: 287267:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10176.337983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10180.941757] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10190.365465] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10190.527209] Lustre: 288136:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10192.533746] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10195.170299] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10203.431321] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10206.815770] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10206.819604] Lustre: Skipped 1 previous similar message [10220.708168] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 00:50:05 (1788583805) [10223.589374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10223.595215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10223.604493] Lustre: Skipped 6 previous similar messages [10223.624134] Lustre: Skipped 10 previous similar messages [10228.671149] Lustre: server umount lustre-MDT0000 complete [10228.709412] LustreError: 285641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10228.724830] LustreError: 285641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10231.707385] LustreError: 285626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583818 with bad export cookie 4660910406162261194 [10231.711406] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10231.717932] LustreError: 285626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10232.033937] Lustre: server umount lustre-MDT0001 complete [10244.325746] Lustre: server umount lustre-OST0000 complete [10257.296104] Lustre: server umount lustre-OST0001 complete [10265.421687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10265.875410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10265.879996] Lustre: Skipped 5 previous similar messages [10270.322401] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10277.963403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10282.452523] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10289.446873] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10294.323736] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10300.338184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10307.195972] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10315.513987] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10330.203499] loop1: detected capacity change from 0 to 204800000 [10330.424890] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10330.433729] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10330.453899] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10335.726455] loop1: detected capacity change from 0 to 204800000 [10341.452688] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10341.534730] LustreError: 290372:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs17637: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10341.546916] LustreError: 290372:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs17637 [10341.551099] LustreError: 290372:0:(mgs_handler.c:517:mgs_target_reg()) fs17637-MDT0001: Error checking target index: rc = -13 [10341.560529] LustreError: fs17637-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10341.568404] LustreError: 294098:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [10341.576086] LustreError: 294098:0:(tgt_mount.c:2132:server_put_super()) no obd fs17637-MDT0001 [10341.581568] LustreError: 294098:0:(tgt_mount.c:116:server_deregister_mount()) fs17637-MDT0001 not registered [10341.626398] Lustre: server umount fs17637-MDT0001 complete [10341.633199] LustreError: 294098:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10350.329528] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10350.461489] Lustre: Setting parameter fs17637-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs17637-MDT0001 [10350.538177] Lustre: srv-fs17637-MDT0001: No data found on store. Initialize space. [10350.548708] Lustre: Skipped 4 previous similar messages [10350.646824] Lustre: fs17637-MDT0001: new disk, initializing [10350.649661] Lustre: Skipped 3 previous similar messages [10350.784722] Lustre: srv-fs17637-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10350.795236] Lustre: 294564:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17637-MDT0001-mdtlov: can't create new llog: rc = -115 [10350.804145] LustreError: 294564:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17637-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10355.500220] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10362.069668] Lustre: server umount fs17637-MDT0001 complete [10375.960798] loop1: detected capacity change from 0 to 204800000 [10382.446076] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10382.972348] Lustre: 296132:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17637-MDT0001-mdtlov: can't create new llog: rc = -115 [10382.984592] LustreError: 296132:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17637-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10386.931873] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10387.473654] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10393.194905] Lustre: server umount fs17637-MDT0001 complete [10406.372917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10406.386645] LustreError: Skipped 1 previous similar message [10406.393303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10406.423690] Lustre: Skipped 1 previous similar message [10406.431891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10411.323400] Lustre: server umount lustre-MDT0000 complete [10421.258881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10421.336818] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10426.477777] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10426.884279] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10435.332201] loop1: detected capacity change from 0 to 204800000 [10441.116461] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10441.569130] Lustre: 298581:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17637-MDT0001-mdtlov: can't create new llog: rc = -115 [10441.581425] LustreError: 298581:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17637-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10445.895504] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10450.400715] Lustre: server umount fs17637-MDT0001 complete [10472.937298] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10472.954802] LustreError: Skipped 1 previous similar message [10486.240197] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10486.491922] Lustre: server umount lustre-OST0000 complete [10497.389174] Lustre: server umount lustre-MDT0000 complete [10498.530127] LustreError: 291443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10498.538344] LustreError: 291443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 111 previous similar messages [10501.114983] LustreError: 300175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584088 with bad export cookie 4660910406162263875 [10501.121574] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10501.136862] LustreError: 300175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10501.609520] Lustre: server umount lustre-MDT0001 complete [10511.747500] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [10514.649279] Key type lgssc unregistered [10514.999759] LNet: 300925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10515.031082] LNetError: 300925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10515.047074] LNet: Removed LNI 192.168.203.129@tcp [10515.973175] Key type .llcrypt unregistered [10515.981135] Key type ._llcrypt unregistered [10535.953363] Key type ._llcrypt registered [10535.955279] Key type .llcrypt registered [10536.072841] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [10537.269667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10537.358983] alg: No test for adler32 (adler32-zlib) [10538.616379] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [10538.827458] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [10540.528411] Key type lgssc registered [10541.685632] Lustre: Echo OBD driver; http://www.lustre.org/ [10549.329202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10549.339739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10550.819403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10555.577436] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10561.000785] LustreError: 302192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10561.029476] LustreError: 302192:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10564.312371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10564.750400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10568.327563] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10574.732841] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10580.036414] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10587.149966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10587.603456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10589.611929] LustreError: 304023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10593.349317] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10594.793203] LustreError: 304021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10600.757942] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10600.931164] LustreError: 304023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10600.961824] LustreError: 304023:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10606.064941] LustreError: 304021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10606.105035] LustreError: 304021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10616.295550] LustreError: 304021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10616.326525] LustreError: 304021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10619.836392] loop1: detected capacity change from 0 to 204800000 [10620.086477] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10620.122072] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10620.144217] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10625.156045] loop1: detected capacity change from 0 to 204800000 [10630.710170] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10630.863948] Lustre: fs28400-OST0000: new disk, initializing [10630.868940] Lustre: srv-fs28400-OST0000: No data found on store. Initialize space. [10630.875373] Lustre: 305927:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10630.977388] Lustre: fs28400-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10634.265818] LustreError: 304024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10634.289155] LustreError: 304024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [10637.750733] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10645.735891] Lustre: server umount fs28400-OST0000 complete [10657.505794] loop1: detected capacity change from 0 to 204800000 [10662.984096] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10663.238715] Lustre: fs28400-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10667.526065] LustreError: 304023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10667.540758] LustreError: 304023:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [10668.965078] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10672.633923] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10677.740701] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10678.512602] loop2: detected capacity change from 0 to 204800000 [10678.901059] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10678.946710] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10678.975727] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10685.129847] loop2: detected capacity change from 0 to 204800000 [10691.172059] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10691.307676] LustreError: fs28400-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10691.327084] LustreError: fs28400-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10691.335744] LustreError: 308967:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [10691.341661] LustreError: 308967:0:(tgt_mount.c:2132:server_put_super()) no obd fs28400-OST0001 [10691.349887] LustreError: 308967:0:(tgt_mount.c:116:server_deregister_mount()) fs28400-OST0001 not registered [10691.391347] Lustre: server umount fs28400-OST0001 complete [10691.396356] LustreError: 308967:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10702.172800] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10702.348628] Lustre: fs28400-OST0001: new disk, initializing [10702.353479] Lustre: srv-fs28400-OST0001: No data found on store. Initialize space. [10702.359574] Lustre: 309518:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10702.447536] Lustre: fs28400-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10708.618092] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10717.062704] Lustre: server umount fs28400-OST0000 complete [10726.192933] Lustre: server umount fs28400-OST0001 complete [10731.554938] LustreError: 304023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10731.576471] LustreError: 304023:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 36 previous similar messages [10738.517477] loop1: detected capacity change from 0 to 204800000 [10744.147414] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10744.487825] Lustre: fs28400-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10751.390330] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10757.706846] loop2: detected capacity change from 0 to 204800000 [10762.692165] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10769.324966] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10777.591489] Lustre: server umount fs28400-OST0000 complete [10785.935839] Lustre: server umount fs28400-OST0001 complete [10799.586803] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10799.592749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10799.619765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10800.628676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10800.629659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10804.892942] Lustre: server umount lustre-OST0000 complete [10810.338732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10810.352934] LustreError: Skipped 1 previous similar message [10810.362660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10810.380579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10810.387176] Lustre: Skipped 1 previous similar message [10814.982595] Lustre: server umount lustre-MDT0000 complete [10818.591767] LustreError: 302173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584405 with bad export cookie 10827408145955175409 [10818.602628] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10818.616245] LustreError: 302173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10818.999641] Lustre: server umount lustre-MDT0001 complete [10828.139749] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [10830.891371] Key type lgssc unregistered [10831.150681] LNet: 315440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10831.159819] LNetError: 315440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10831.187942] LNet: Removed LNI 192.168.203.129@tcp [10832.183292] Key type .llcrypt unregistered [10832.187921] Key type ._llcrypt unregistered [10863.992604] Key type ._llcrypt registered [10863.996925] Key type .llcrypt registered [10864.091686] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [10883.098488] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [10884.530468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10884.660530] alg: No test for adler32 (adler32-zlib) [10886.015418] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [10886.434160] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [10888.152349] Key type lgssc registered [10889.326709] Lustre: Echo OBD driver; http://www.lustre.org/ [10922.360600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10922.395119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10923.687903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10923.736867] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10923.866165] Lustre: lustre-MDT0000: new disk, initializing [10923.991303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10924.011488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10928.335705] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10939.329057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10939.458582] Lustre: 319308:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10939.527697] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10939.538403] Lustre: Skipped 1 previous similar message [10939.626141] Lustre: lustre-MDT0001: new disk, initializing [10939.753032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10939.809943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10939.832915] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10944.216940] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10952.351057] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10957.535668] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10963.664121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10964.082187] Lustre: lustre-OST0000: new disk, initializing [10964.085971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10964.093136] Lustre: 321278:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10964.226420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10969.996688] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10974.274192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10974.290249] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10974.437435] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10978.652680] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10980.537693] Lustre: Mounted lustre-client - version 2.17.58_4_g27fe0b7 [10981.374559] Lustre: Unmounted lustre-client [10991.078090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10991.099695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10991.102822] Lustre: Skipped 1 previous similar message [10991.132840] Lustre: Skipped 1 previous similar message [10996.203051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10996.217202] Lustre: Skipped 1 previous similar message [10996.742291] Lustre: server umount lustre-OST0000 complete [11001.320270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11001.326693] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11001.349424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11001.366644] Lustre: Skipped 1 previous similar message [11006.434484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11006.440969] Lustre: Skipped 2 previous similar messages [11011.554591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11011.570523] Lustre: Skipped 1 previous similar message [11015.137228] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11015.345989] Lustre: server umount lustre-MDT0000 complete [11018.935992] LustreError: 319300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584605 with bad export cookie 3812363407668792082 [11018.950622] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11018.958659] LustreError: 319300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11019.291811] Lustre: server umount lustre-MDT0001 complete [11030.196806] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [11033.447488] Key type lgssc unregistered [11042.082206] LNet: 1 local NIs in recovery (showing 1): 192.168.203.129@tcp [11043.040966] LNet: 323439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11043.052315] LNetError: 323439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11109.606726] LNet: Removed LNI 192.168.203.129@tcp [11110.476073] Key type .llcrypt unregistered [11110.478656] Key type ._llcrypt unregistered [11129.300684] Key type ._llcrypt registered [11129.307071] Key type .llcrypt registered [11129.478328] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [11145.495967] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [11146.462860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11146.488304] alg: No test for adler32 (adler32-zlib) [11147.816722] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [11148.082097] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [11149.808175] Key type lgssc registered [11150.980408] Lustre: Echo OBD driver; http://www.lustre.org/ [11181.143339] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 01:06:05 (1788584765) [11187.717301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11187.760657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11189.070662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11189.095857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11189.175585] Lustre: lustre-MDT0000: new disk, initializing [11189.308757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11189.322386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11193.768297] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11204.857659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11205.131065] Lustre: 326655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11205.254947] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11205.259443] Lustre: Skipped 1 previous similar message [11205.408571] Lustre: lustre-MDT0001: new disk, initializing [11205.571739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11205.637918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11205.659751] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11211.717316] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11222.397958] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11228.828722] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11236.142793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11236.476423] Lustre: lustre-OST0000: new disk, initializing [11236.479924] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11236.487842] Lustre: 328624:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11236.635123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11243.606052] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11245.148647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11245.165288] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11245.294260] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11255.017221] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11276.268419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11276.273970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11276.284855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11276.287921] Lustre: Skipped 1 previous similar message [11276.313863] LustreError: Skipped 1 previous similar message [11280.198765] Lustre: server umount lustre-OST0000 complete [11286.504099] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11286.529449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11286.543055] Lustre: Skipped 2 previous similar messages [11291.617284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11291.623530] Lustre: Skipped 1 previous similar message [11296.737738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11296.743963] Lustre: Skipped 1 previous similar message [11299.296377] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11299.547595] Lustre: server umount lustre-MDT0000 complete [11303.824380] LustreError: 328623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584890 with bad export cookie 2825807414348020094 [11303.826720] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11303.841709] LustreError: 328623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11304.139730] Lustre: server umount lustre-MDT0001 complete [11318.097712] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [11322.034416] Key type lgssc unregistered [11322.481548] LNet: 330534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11322.494614] LNetError: 330534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11322.538801] LNet: Removed LNI 192.168.203.129@tcp [11323.639824] Key type .llcrypt unregistered [11323.646915] Key type ._llcrypt unregistered [11348.690910] Key type ._llcrypt registered [11348.693914] Key type .llcrypt registered [11348.791980] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [11349.749682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11349.912163] alg: No test for adler32 (adler32-zlib) [11350.981169] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [11351.223302] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [11352.944183] Key type lgssc registered [11354.032190] Lustre: Echo OBD driver; http://www.lustre.org/ [11362.212546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11362.240376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11363.674897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11368.632215] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11374.058737] LustreError: 331805:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11374.082777] LustreError: 331805:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11377.563794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11378.112042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11382.923888] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11390.697519] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11397.610762] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11406.125670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11406.673410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11414.022872] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11423.090381] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11433.400097] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11435.256433] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11441.251791] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.29@tcp (stopping) [11443.700051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11443.702790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11443.716357] Lustre: Skipped 2 previous similar messages [11443.737790] Lustre: Skipped 2 previous similar messages [11446.298751] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.29@tcp (stopping) [11446.749414] Lustre: server umount lustre-MDT0000 complete [11448.809288] LustreError: 332806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11448.836328] LustreError: 332806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11450.607040] LustreError: 331787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585037 with bad export cookie 721744567579234665 [11450.614487] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11450.627836] LustreError: 331787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11451.419498] LustreError: 331802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11451.477410] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.29@tcp (stopping) [11453.923877] LustreError: 331801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11453.925251] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11455.973015] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11455.978722] Lustre: Skipped 1 previous similar message [11457.199544] Lustre: server umount lustre-MDT0001 complete [11461.289154] Lustre: server umount lustre-OST0000 complete [11470.426285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11476.001366] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11479.334292] Lustre: 335985:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.129@tcp [11481.619603] Lustre: server umount MGS complete [11490.702807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11491.306306] LustreError: 336558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11491.323171] LustreError: 336558:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11491.371599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11496.016603] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11501.549334] LustreError: 336558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11501.576334] LustreError: 336558:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11505.653280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11506.221229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11511.429659] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11521.386705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11521.964575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11529.595555] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11537.497622] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11539.313534] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11545.257852] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11546.990796] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11551.715460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11551.720209] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11551.725522] LustreError: Skipped 1 previous similar message [11551.726719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11551.726725] Lustre: Skipped 1 previous similar message [11551.750525] Lustre: Skipped 1 previous similar message [11553.791704] Lustre: server umount lustre-OST0000 complete [11561.976184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11561.992857] Lustre: Skipped 1 previous similar message [11564.339736] Lustre: server umount lustre-MDT0000 complete [11567.074665] LustreError: 337287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11567.104482] LustreError: 337287:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11568.910067] LustreError: 339041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585155 with bad export cookie 721744567579235498 [11568.925449] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11568.927959] LustreError: 339041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11569.682776] Lustre: server umount lustre-MDT0001 complete [11587.357751] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [11590.510322] Key type lgssc unregistered [11590.881562] LNet: 340016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11590.888407] LNetError: 340016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11590.919787] LNet: Removed LNI 192.168.203.129@tcp [11592.235460] Key type .llcrypt unregistered [11592.237763] Key type ._llcrypt unregistered [11615.426482] Key type ._llcrypt registered [11615.443330] Key type .llcrypt registered [11615.628852] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [11634.493631] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [11636.244318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11636.402046] alg: No test for adler32 (adler32-zlib) [11637.503953] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [11637.840784] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [11639.712230] Key type lgssc registered [11641.079343] Lustre: Echo OBD driver; http://www.lustre.org/ [11679.925624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11679.981735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11681.385412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11681.481128] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11681.609126] Lustre: lustre-MDT0000: new disk, initializing [11681.727962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11681.776684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11688.369506] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11701.952903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11702.065333] Lustre: 343178:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11702.117590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11702.125138] Lustre: Skipped 1 previous similar message [11702.191377] Lustre: lustre-MDT0001: new disk, initializing [11702.306695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11702.350302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11702.365374] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11707.020165] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11716.929372] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11723.477921] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11731.767670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11732.088910] Lustre: lustre-OST0000: new disk, initializing [11732.093669] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11732.108595] Lustre: 345145:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11732.247512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11738.835512] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11741.272815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11741.282901] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11741.365649] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11748.983489] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11753.863746] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11754.057475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11758.427744] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11758.603609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11762.146308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11762.148868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11762.152912] LustreError: Skipped 1 previous similar message [11762.157916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11762.187274] Lustre: Skipped 1 previous similar message [11766.694160] Lustre: server umount lustre-OST0000 complete [11772.387233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11772.389361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11772.404424] Lustre: Skipped 1 previous similar message [11772.416234] Lustre: Skipped 2 previous similar messages [11776.432286] Lustre: server umount lustre-MDT0000 complete [11779.849819] LustreError: 343983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585366 with bad export cookie 13752818704209874296 [11779.861541] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11779.866332] LustreError: 343983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11780.321206] Lustre: server umount lustre-MDT0001 complete [11789.392196] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 01:16:13 (1788585373) [11807.566574] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [11825.100233] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [11871.744623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11872.178961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11872.237416] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11872.371262] Lustre: lustre-MDT0000: new disk, initializing [11872.622848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11872.652167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11879.820547] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11893.673984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11893.858994] Lustre: 349928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11893.976595] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11893.983882] Lustre: Skipped 1 previous similar message [11894.156549] Lustre: lustre-MDT0001: new disk, initializing [11894.358045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11894.402456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11894.420806] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11900.187957] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11911.871637] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11917.879613] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11923.921566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11924.213663] Lustre: lustre-OST0000: new disk, initializing [11924.228937] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11924.239062] Lustre: 351900:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11924.384036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11931.134542] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11931.715984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11931.731329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11931.851703] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11940.496986] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11978.212158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11978.215065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11978.229262] LustreError: Skipped 1 previous similar message [11978.231272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11978.292085] Lustre: Skipped 1 previous similar message [11982.627396] Lustre: server umount lustre-OST0000 complete [11987.435331] LustreError: 341397:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000030f75be4 x1875467539939072/t0(0) o41->lustre-MDT0001-osp-MDT0000@0@lo:24/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11988.451206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11988.454950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11988.479091] Lustre: Skipped 1 previous similar message [11988.503842] Lustre: Skipped 2 previous similar messages [11993.570649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11993.584913] Lustre: Skipped 1 previous similar message [11993.992619] Lustre: server umount lustre-MDT0000 complete [11998.691222] LustreError: 349940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11998.716444] LustreError: 349940:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11998.855303] LustreError: 349920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585585 with bad export cookie 13752818704209875185 [11998.866482] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11999.295864] Lustre: server umount lustre-MDT0001 complete [12013.695796] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [12016.651076] Key type lgssc unregistered [12017.000660] LNet: 354072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12017.022961] LNetError: 354072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12017.057542] LNet: Removed LNI 192.168.203.129@tcp [12018.395628] Key type .llcrypt unregistered [12018.397754] Key type ._llcrypt unregistered [12127.922362] Key type ._llcrypt registered [12127.924810] Key type .llcrypt registered [12128.011333] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [12128.358376] Key type .llcrypt unregistered [12128.364783] Key type ._llcrypt unregistered [12212.427150] Key type ._llcrypt registered [12212.428986] Key type .llcrypt registered [12212.539152] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [12212.806807] Key type .llcrypt unregistered [12212.812856] Key type ._llcrypt unregistered [12241.840891] Key type ._llcrypt registered [12241.847376] Key type .llcrypt registered [12241.991437] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [12242.281768] Key type .llcrypt unregistered [12242.288071] Key type ._llcrypt unregistered [12271.550304] Key type ._llcrypt registered [12271.552392] Key type .llcrypt registered [12271.651320] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [12271.900377] Key type .llcrypt unregistered [12271.906925] Key type ._llcrypt unregistered [12348.497402] Key type ._llcrypt registered [12348.507650] Key type .llcrypt registered [12348.641358] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [12349.009606] Key type .llcrypt unregistered [12349.013706] Key type ._llcrypt unregistered [12385.269316] Key type ._llcrypt registered [12385.271835] Key type .llcrypt registered [12385.402471] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [12385.675620] Key type .llcrypt unregistered [12385.677928] Key type ._llcrypt unregistered [12426.359628] Key type ._llcrypt registered [12426.362140] Key type .llcrypt registered [12426.507460] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [12427.061127] Key type .llcrypt unregistered [12427.067964] Key type ._llcrypt unregistered [12460.960085] Key type ._llcrypt registered [12460.964764] Key type .llcrypt registered [12461.107170] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [12461.484539] Key type .llcrypt unregistered [12461.487241] Key type ._llcrypt unregistered [12492.349903] Key type ._llcrypt registered [12492.352755] Key type .llcrypt registered [12492.480636] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [12492.835322] Key type .llcrypt unregistered [12492.837561] Key type ._llcrypt unregistered [12523.570101] Key type ._llcrypt registered [12523.573594] Key type .llcrypt registered [12523.779810] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [12524.146378] Key type .llcrypt unregistered [12524.148684] Key type ._llcrypt unregistered [12589.245289] Key type ._llcrypt registered [12589.248360] Key type .llcrypt registered [12589.374371] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [12589.764502] Key type .llcrypt unregistered [12589.769564] Key type ._llcrypt unregistered [12620.889214] Key type ._llcrypt registered [12620.894849] Key type .llcrypt registered [12621.023338] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [12621.333629] Key type .llcrypt unregistered [12621.338694] Key type ._llcrypt unregistered [12660.760716] Key type ._llcrypt registered [12660.764761] Key type .llcrypt registered [12660.881771] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [12661.252405] Key type .llcrypt unregistered [12661.256715] Key type ._llcrypt unregistered [12720.721161] Key type ._llcrypt registered [12720.725995] Key type .llcrypt registered [12720.847562] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [12737.585383] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [12739.139626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12739.219122] alg: No test for adler32 (adler32-zlib) [12740.625217] Lustre: Lustre: Build Version: 2.17.58_4_g27fe0b7 [12741.022389] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [12742.784340] Key type lgssc registered [12743.993418] Lustre: Echo OBD driver; http://www.lustre.org/ [12785.692618] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12785.736213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12787.150500] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12787.237954] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12787.387517] Lustre: tst250fs-MDT0000: new disk, initializing [12787.454629] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12787.486558] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12792.604277] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12802.998103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12803.162535] Lustre: 374372:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12803.216642] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12803.220419] Lustre: Skipped 1 previous similar message [12803.285583] Lustre: tst250fs-MDT0001: new disk, initializing [12803.419179] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12803.469987] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12803.492398] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12808.025173] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12812.450371] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12819.119785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12819.418645] Lustre: tst250fs-OST0000: new disk, initializing [12819.429286] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12819.438893] Lustre: 376007:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12819.551136] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12822.405798] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12822.414867] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12822.502728] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12825.493447] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12836.040810] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12836.307651] Lustre: tst250fs-OST0001: new disk, initializing [12836.315575] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12836.332580] Lustre: 376880:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12836.531164] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12843.176951] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12843.185718] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12843.350777] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12844.883408] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12851.720293] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12854.053462] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [12854.062278] Lustre: Skipped 1 previous similar message [12874.214453] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12874.218675] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12874.233459] Lustre: Skipped 3 previous similar messages [12874.236772] Lustre: Skipped 2 previous similar messages [12878.846380] Lustre: server umount tst250fs-MDT0000 complete [12881.971804] LustreError: 376017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586468 with bad export cookie 17493141476773904215 [12881.980477] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12881.981717] LustreError: 376017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12882.439086] Lustre: server umount tst250fs-MDT0001 complete [12897.189226] Lustre: server umount tst250fs-OST0000 complete [12910.963638] Lustre: server umount tst250fs-OST0001 complete [12928.128192] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [12935.729556] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [12968.353591] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 01:35:52 (1788586552) [12970.452763] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12985.853680] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [12994.569516] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [13023.938796] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12759 sec ========= 01:36:48 (1788586608) [13025.485939] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:36:49 (1788586609) === [13029.284586] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:36:53 (1788586613) === [13072.484174] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [13076.456863] Key type lgssc unregistered [13076.801897] LNet: 386005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13076.821170] LNetError: 386005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13076.849889] LNet: Removed LNI 192.168.203.129@tcp [13077.797575] Key type .llcrypt unregistered [13077.799854] Key type ._llcrypt unregistered