[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 515447067 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002373] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007421] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009026] pid_max: default: 32768 minimum: 301 [ 0.010153] LSM: Security Framework initializing [ 0.011074] Yama: becoming mindful. [ 0.012048] SELinux: Initializing. [ 0.013087] *** VALIDATE selinux *** [ 0.022206] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026794] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028102] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030108] *** VALIDATE tmpfs *** [ 0.031541] *** VALIDATE proc *** [ 0.032000] *** VALIDATE cgroup *** [ 0.032011] *** VALIDATE cgroup2 *** [ 0.034214] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038026] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.041984] debug: unmapping init [mem 0xffffffff88059000-0xffffffff88060fff] [ 0.044923] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045696] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046021] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050011] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.051988] ... event mask: 000000070000000f [ 0.052327] rcu: Hierarchical SRCU implementation. [ 0.054463] smp: Bringing up secondary CPUs ... [ 0.055658] x86: Booting SMP configuration: [ 0.056024] .... node #0, CPUs: #1 #2 #3 [ 0.059499] smp: Brought up 1 node, 4 CPUs [ 0.061009] smpboot: Max logical packages: 1 [ 0.061887] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135709] node 0 deferred pages initialised in 70ms [ 0.138173] devtmpfs: initialized [ 0.139235] x86/mm: Memory block size: 128MB [ 0.141900] gcov: version magic: 0x41383552 [ 0.143280] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.144117] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.145332] pinctrl core: initialized pinctrl subsystem [ 0.146228] [ 0.146927] ************************************************************* [ 0.147020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.148015] ** ** [ 0.149011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.150016] ** ** [ 0.151016] ** This means that this kernel is built to expose internal ** [ 0.152014] ** IOMMU data structures, which may compromise security on ** [ 0.153014] ** your system. ** [ 0.154016] ** ** [ 0.155018] ** If you see this message and you are not debugging the ** [ 0.156014] ** kernel, report this immediately to your vendor! ** [ 0.157014] ** ** [ 0.158012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159012] ************************************************************* [ 0.160794] NET: Registered protocol family 16 [ 0.161467] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.162075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.163066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.164466] cpuidle: using governor menu [ 0.167259] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.169485] PCI: Using configuration type 1 for base access [ 0.172128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.181057] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.183138] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.187248] cryptd: max_cpu_qlen set to 1000 [ 0.190789] ACPI: Added _OSI(Module Device) [ 0.192010] ACPI: Added _OSI(Processor Device) [ 0.194017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.195012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.200206] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.206587] ACPI: Interpreter enabled [ 0.208058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.210013] ACPI: Using IOAPIC for interrupt routing [ 0.211103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.214457] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.223588] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.226103] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.228019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.232102] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.236369] acpiphp: Slot [2] registered [ 0.238162] acpiphp: Slot [5] registered [ 0.240165] acpiphp: Slot [6] registered [ 0.241130] acpiphp: Slot [7] registered [ 0.242104] acpiphp: Slot [8] registered [ 0.244156] acpiphp: Slot [9] registered [ 0.245104] acpiphp: Slot [10] registered [ 0.247126] acpiphp: Slot [3] registered [ 0.248094] acpiphp: Slot [4] registered [ 0.250136] acpiphp: Slot [11] registered [ 0.251119] acpiphp: Slot [12] registered [ 0.252074] acpiphp: Slot [13] registered [ 0.254101] acpiphp: Slot [14] registered [ 0.255108] acpiphp: Slot [15] registered [ 0.257122] acpiphp: Slot [16] registered [ 0.258060] acpiphp: Slot [17] registered [ 0.260113] acpiphp: Slot [18] registered [ 0.261095] acpiphp: Slot [19] registered [ 0.263120] acpiphp: Slot [20] registered [ 0.264103] acpiphp: Slot [21] registered [ 0.266120] acpiphp: Slot [22] registered [ 0.267083] acpiphp: Slot [23] registered [ 0.269153] acpiphp: Slot [24] registered [ 0.270117] acpiphp: Slot [25] registered [ 0.272100] acpiphp: Slot [26] registered [ 0.273109] acpiphp: Slot [27] registered [ 0.275110] acpiphp: Slot [28] registered [ 0.276097] acpiphp: Slot [29] registered [ 0.278125] acpiphp: Slot [30] registered [ 0.279119] acpiphp: Slot [31] registered [ 0.281055] PCI host bridge to bus 0000:00 [ 0.282013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.284021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.287023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.289032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.291026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.293023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.295173] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.298008] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.301247] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.311020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.316000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.319025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.321018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.323016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.326532] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.328829] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.331047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.335106] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.341019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.356017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.361015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.367606] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.379019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.389018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.409020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.418136] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.425019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.432018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.447017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.456829] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.464020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.471016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.491021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.503858] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.512015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.520016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.540026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.551343] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.559019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.564017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.580020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.593982] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.604028] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.612033] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.629017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.639152] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.641319] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.643361] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.645265] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.647229] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.652139] iommu: Default domain type: Passthrough [ 0.654469] SCSI subsystem initialized [ 0.657162] ACPI: bus type USB registered [ 0.658098] usbcore: registered new interface driver usbfs [ 0.659267] usbcore: registered new interface driver hub [ 0.661102] usbcore: registered new device driver usb [ 0.663194] pps_core: LinuxPPS API ver. 1 registered [ 0.664008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.667062] PTP clock support registered [ 0.668211] EDAC MC: Ver: 3.0.0 [ 0.669502] PCI: Using ACPI for IRQ routing [ 0.671087] NetLabel: Initializing [ 0.672000] NetLabel: domain hash size = 128 [ 0.674023] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.676195] NetLabel: unlabeled traffic allowed by default [ 0.679106] vgaarb: loaded [ 0.680275] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.682011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.688422] clocksource: Switched to clocksource kvm-clock [ 0.797402] VFS: Disk quotas dquot_6.6.0 [ 0.798855] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.801306] *** VALIDATE ramfs *** [ 0.802511] *** VALIDATE hugetlbfs *** [ 0.804594] pnp: PnP ACPI init [ 0.807066] pnp: PnP ACPI: found 6 devices [ 0.832285] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.836177] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.838446] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.840626] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.843098] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.845461] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.848114] NET: Registered protocol family 2 [ 0.850560] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.854592] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.857822] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.862752] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.866141] TCP: Hash tables configured (established 65536 bind 65536) [ 0.869050] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.871918] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.874614] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.877423] NET: Registered protocol family 1 [ 0.879884] RPC: Registered named UNIX socket transport module. [ 0.882446] RPC: Registered udp transport module. [ 0.884479] RPC: Registered tcp transport module. [ 0.886321] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.888971] NET: Registered protocol family 44 [ 0.890743] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.892864] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.895031] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.897561] PCI: CLS 0 bytes, default 64 [ 0.899265] Unpacking initramfs... [ 2.350103] debug: unmapping init [mem 0xffff9ae97cc54000-0xffff9ae97ffbffff] [ 2.354127] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.356166] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.358672] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.880358] Initialise system trusted keyrings [ 2.882254] Key type blacklist registered [ 2.884155] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.893820] zbud: loaded [ 2.897415] *** VALIDATE nfs *** [ 2.898765] *** VALIDATE nfs4 *** [ 2.900551] pstore: using deflate compression [ 2.904676] Platform Keyring initialized [ 3.011457] NET: Registered protocol family 38 [ 3.013089] Key type asymmetric registered [ 3.014500] Asymmetric key parser 'x509' registered [ 3.016118] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.018806] io scheduler mq-deadline registered [ 3.020315] io scheduler kyber registered [ 3.021836] io scheduler bfq registered [ 3.023525] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.026388] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.028971] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.031650] ACPI: Power Button [PWRF] [ 3.037044] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.043584] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.058412] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.065669] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.087989] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.119749] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.151968] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.156358] Non-volatile memory driver v1.3 [ 3.157815] Linux agpgart interface v0.103 [ 3.186791] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.189219] vda: detected capacity change from 0 to 76779520 [ 3.205379] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.207811] vdb: detected capacity change from 0 to 1073741824 [ 3.223436] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.226050] vdc: detected capacity change from 0 to 2621440000 [ 3.242439] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.245062] vdd: detected capacity change from 0 to 2621440000 [ 3.259617] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.262335] vde: detected capacity change from 0 to 4294967296 [ 3.277654] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.280615] vdf: detected capacity change from 0 to 4294967296 [ 3.292584] libphy: Fixed MDIO Bus: probed [ 3.297782] usbcore: registered new interface driver usbserial_generic [ 3.300118] usbserial: USB Serial support registered for generic [ 3.302246] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.306666] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.308286] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.310448] mousedev: PS/2 mouse device common for all mice [ 3.314204] rtc_cmos 00:05: RTC can wake from S4 [ 3.316572] rtc_cmos 00:05: registered as rtc0 [ 3.318570] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.320941] intel_pstate: CPU model not supported [ 3.324264] hid: raw HID events driver (C) Jiri Kosina [ 3.326644] usbcore: registered new interface driver usbhid [ 3.326832] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.328674] usbhid: USB HID core driver [ 3.328852] drop_monitor: Initializing network drop monitor service [ 3.336603] Initializing XFRM netlink socket [ 3.338479] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.339652] NET: Registered protocol family 10 [ 3.344120] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.344836] Segment Routing with IPv6 [ 3.348760] NET: Registered protocol family 17 [ 3.350955] mpls_gso: MPLS GSO support [ 3.359561] RAS: Correctable Errors collector initialized. [ 3.361478] AVX version of gcm_enc/dec engaged. [ 3.363037] AES CTR mode by8 optimization enabled [ 3.435926] sched_clock: Marking stable (3435891441, 0)->(4384706245, -948814804) [ 3.439135] registered taskstats version 1 [ 3.441314] Loading compiled-in X.509 certificates [ 3.443231] zswap: loaded using pool lzo/zbud [ 3.467937] Key type big_key registered [ 3.480421] Key type encrypted registered [ 3.481954] ima: No TPM chip found, activating TPM-bypass! [ 3.484079] ima: Allocated hash algorithm: sha1 [ 3.485563] ima: No architecture policies found [ 3.487219] evm: Initialising EVM extended attributes: [ 3.488936] evm: security.selinux [ 3.490218] evm: security.ima [ 3.491259] evm: security.capability [ 3.492493] evm: HMAC attrs: 0x1 [ 3.494687] rtc_cmos 00:05: setting system clock to 2026-09-13 10:52:03 UTC (1789296723) [ 3.500774] debug: unmapping init [mem 0xffffffff89003000-0xffffffff891fffff] [ 3.503904] debug: unmapping init [mem 0xffffffff87d82000-0xffffffff88058fff] [ 3.515082] Write protecting the kernel read-only data: 28672k [ 3.518834] debug: unmapping init [mem 0xffffffff86403000-0xffffffff865fffff] [ 3.522241] debug: unmapping init [mem 0xffffffff86d14000-0xffffffff86dfffff] [ 3.557600] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.566957] systemd[1]: Detected virtualization kvm. [ 3.568818] systemd[1]: Detected architecture x86-64. [ 3.571157] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.596376] systemd[1]: No hostname configured. [ 3.598200] systemd[1]: Set hostname to . [ 3.600131] random: systemd: uninitialized urandom read (16 bytes read) [ 3.602713] systemd[1]: Initializing machine ID from random generator. [ 3.643466] random: ln: uninitialized urandom read (6 bytes read) [ 3.734159] random: systemd: uninitialized urandom read (16 bytes read) [ 3.736705] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.741744] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.746371] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.314835] device-mapper: uevent: version 1.0.3 [ 4.317128] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.957254] virtio_net virtio0 ens2: renamed from eth0 [ 4.975254] random: fast init done [ 5.039346] scsi host0: ata_piix [ 5.065337] scsi host1: ata_piix [ 5.066779] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.068982] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.315596] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.997334] random: crng init done [ 9.998793] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.317322] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.468677] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.758176] SELinux: Disabled at runtime. [ 11.821230] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.828910] systemd[1]: Detected virtualization kvm. [ 11.830763] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.351840] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.355282] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.359686] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.363274] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.366787] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.374107] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.379180] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 12.530585] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.793927] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.029227] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.084628] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.231261] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.242900] EDAC sbridge: Ver: 1.1.2 [ 15.106704] Key type dns_resolver registered [ 15.415624] NFS: Registering the id_resolver key type [ 15.417561] Key type id_resolver registered [ 15.419149] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg457-server login: [ 44.279704] libcfs: loading out-of-tree module taints kernel. [ 44.361128] Key type ._llcrypt registered [ 44.362908] Key type .llcrypt registered [ 44.499765] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 68.570630] hrtimer: interrupt took 10611220 ns [ 68.842652] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 70.920334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 70.933494] alg: No test for adler32 (adler32-zlib) [ 72.929243] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 74.764109] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 76.793596] Key type lgssc registered [ 80.100119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 100.751845] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 149.037660] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 163.797146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 163.844648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 165.286160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 165.336933] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 165.489969] Lustre: lustre-MDT0000: new disk, initializing [ 165.614854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 165.652020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 171.047850] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 186.628498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 186.756615] Lustre: 6502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 186.822784] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 186.832962] Lustre: Skipped 1 previous similar message [ 187.009681] Lustre: lustre-MDT0001: new disk, initializing [ 187.076182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 187.125822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 187.142921] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 192.802806] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.895319] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 208.251141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 208.828763] Lustre: lustre-OST0000: new disk, initializing [ 208.843630] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 208.855191] Lustre: 8444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 208.968476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 217.657146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 217.682866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 217.780612] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 219.023376] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.115539] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 236.263333] Lustre: lustre-OST0001: new disk, initializing [ 236.266832] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 236.272853] Lustre: 9517:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 236.351241] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 243.280620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 243.300604] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 243.419517] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 243.589747] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 255.981734] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 261.382327] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 267.226545] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing check_logdir /tmp/testlogs/ [ 273.183568] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing yml_node [ 277.655338] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 280.913846] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 284.253238] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 286.331781] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Sep 13 06:56:44 EDT 2026 [ 307.627921] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 309.213064] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 325.609219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 325.624420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 325.646130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 328.676532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 328.692190] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.097740] Lustre: server umount lustre-MDT0000 complete [ 334.637739] LustreError: 6494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789297054 with bad export cookie 14458898686314731761 [ 334.639151] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 334.647274] LustreError: 6494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 338.916308] LustreError: 8232:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 338.916775] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.924535] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 338.924543] Lustre: Skipped 3 previous similar messages [ 338.938064] LustreError: 8232:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 338.978390] Lustre: Skipped 3 previous similar messages [ 339.938232] LustreError: 6510:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 339.962024] LustreError: 6510:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 341.031055] Lustre: server umount lustre-MDT0001 complete [ 345.136730] Lustre: server umount lustre-OST0000 complete [ 349.243521] Lustre: server umount lustre-OST0001 complete [ 356.836786] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 369.462525] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 414.617684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 414.881904] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 414.996920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 415.137702] Lustre: lustre-MDT0000: new disk, initializing [ 415.306338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 415.330837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 420.791391] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 434.403133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 434.562828] Lustre: 15070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 434.643063] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 434.649792] Lustre: Skipped 1 previous similar message [ 434.720753] Lustre: lustre-MDT0001: new disk, initializing [ 434.833874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 434.866053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 434.892107] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 439.877318] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 450.287743] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 457.311688] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 465.131951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 465.702733] Lustre: lustre-OST0000: new disk, initializing [ 465.709680] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 465.718290] Lustre: 17038:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 465.844379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.680824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 467.688329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 467.842832] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 473.230482] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 484.270432] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 489.708991] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 489.973226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 496.231741] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 496.438130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 498.657776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 498.669672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 498.678148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 498.688527] Lustre: Skipped 1 previous similar message [ 498.702937] Lustre: Skipped 4 previous similar messages [ 503.785568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 503.796399] Lustre: Skipped 1 previous similar message [ 505.125225] Lustre: server umount lustre-OST0000 complete [ 511.975206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 511.986786] LustreError: Skipped 1 previous similar message [ 511.996546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 512.025512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 524.256859] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 524.544607] Lustre: server umount lustre-MDT0000 complete [ 528.353987] LustreError: 15082:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.369519] LustreError: 15082:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 528.580438] LustreError: 15062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789297248 with bad export cookie 14458898686314733294 [ 528.582903] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 528.600839] LustreError: 15062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 529.318189] Lustre: server umount lustre-MDT0001 complete [ 536.000940] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 07:00:53 (1789297253) [ 557.475510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.953326] LustreError: 19929:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.061565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 562.810779] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 563.171484] LustreError: 19930:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.199142] LustreError: 19930:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 572.334175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 578.161994] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 588.380667] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 595.216714] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 597.882824] Lustre: Failing over lustre-MDT0000 [ 598.383176] Lustre: server umount lustre-MDT0000 complete [ 598.502573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 598.517720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 598.538698] Lustre: Skipped 1 previous similar message [ 598.549794] LustreError: 19929:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 598.578916] LustreError: 19929:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 617.965368] LustreError: 21474:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 617.998747] LustreError: 21474:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 618.967914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 618.977537] Lustre: 3634:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789297322/real 1789297322] req@ffff9ae8c366ca80 x1876213675232128/t0(0) o400->MGC192.168.204.157@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789297338 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 619.020905] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 629.217934] LustreError: 3631:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9ae9ea361180 x1876213675234816/t0(0) o250->MGC192.168.204.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 629.553996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 629.565213] Lustre: Skipped 1 previous similar message [ 629.633956] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 634.042548] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 634.860080] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 634.872441] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 634.898333] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 643.715920] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 650.210565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 650.225195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 650.243779] Lustre: Skipped 1 previous similar message [ 650.257050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 650.263902] Lustre: Skipped 6 previous similar messages [ 654.205227] Lustre: server umount lustre-MDT0000 complete [ 657.378353] LustreError: 19930:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 657.403589] LustreError: 19930:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 658.640989] LustreError: 22592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789297378 with bad export cookie 14458898686314734708 [ 658.642377] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 658.653916] LustreError: 22592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 659.118562] Lustre: server umount lustre-MDT0001 complete [ 668.524131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 674.129403] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 683.132671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 687.996328] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 696.097391] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 702.428871] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 705.025478] Lustre: Failing over lustre-MDT0000 [ 705.236807] Lustre: server umount lustre-MDT0000 complete [ 708.142903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 708.163138] Lustre: Skipped 1 previous similar message [ 723.443735] LustreError: 23263:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 723.465298] LustreError: 23263:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 724.449122] Lustre: 3633:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789297428/real 1789297428] req@ffff9ae8c34ac380 x1876213675262208/t0(0) o400->MGC192.168.204.157@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789297444 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 724.484142] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 724.768719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 735.721772] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc8a853bfa926e81e [ 735.748200] Lustre: MGC192.168.204.157@tcp: Connection restored to 0@lo (at 0@lo) [ 735.755720] Lustre: Skipped 1 previous similar message [ 736.085600] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 740.808474] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 741.353725] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 741.364239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 741.376639] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 751.609587] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 756.711166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 756.717517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 756.724305] Lustre: Skipped 2 previous similar messages [ 762.376585] Lustre: server umount lustre-MDT0000 complete [ 766.111446] LustreError: 23250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789297486 with bad export cookie 14458898686314735646 [ 766.119805] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 766.122233] LustreError: 23250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 766.436622] Lustre: server umount lustre-MDT0001 complete [ 775.995809] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 07:04:53 (1789297493) [ 796.131127] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 810.635840] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 854.577187] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 866.350073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 866.624247] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 866.648256] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 866.693288] Lustre: lustre-MDT0000: new disk, initializing [ 866.744695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 866.748479] Lustre: Skipped 3 previous similar messages [ 866.756941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 871.682634] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 885.104585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 885.308593] Lustre: 29668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 885.805691] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 891.569789] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 900.652383] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 906.070465] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 913.962897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 914.227415] Lustre: 31642:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 916.081938] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 916.122928] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 916.244569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 916.262233] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 924.325165] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 935.952417] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 942.207187] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 967.649793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 967.659684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 967.672078] LustreError: Skipped 1 previous similar message [ 967.694955] Lustre: Skipped 3 previous similar messages [ 967.699448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 967.711815] Lustre: Skipped 3 previous similar messages [ 978.402164] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 978.618986] Lustre: server umount lustre-OST0000 complete [ 988.592747] Lustre: server umount lustre-MDT0000 complete [ 992.278441] LustreError: 29660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789297712 with bad export cookie 14458898686314736059 [ 992.290861] LustreError: 29660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 992.294948] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 992.732817] Lustre: server umount lustre-MDT0001 complete [ 1004.561289] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 1007.138029] Key type lgssc unregistered [ 1007.484359] LNet: 33621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.494721] LNetError: 33621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.515992] LNet: Removed LNI 192.168.204.157@tcp [ 1008.335176] Key type .llcrypt unregistered [ 1008.337570] Key type ._llcrypt unregistered [ 1032.280580] Key type ._llcrypt registered [ 1032.285400] Key type .llcrypt registered [ 1032.394153] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 1033.517732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1033.546601] alg: No test for adler32 (adler32-zlib) [ 1034.637136] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 1034.932838] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 1036.648550] Key type lgssc registered [ 1037.678924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1044.905068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1044.946390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1046.479861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1051.574255] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1056.740232] LustreError: 35034:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1056.754928] LustreError: 35034:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1060.954901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1061.299855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1065.796675] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1073.373416] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1080.631887] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1088.409993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1088.907661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1092.041257] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1096.186500] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1105.288280] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1106.368525] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1120.221261] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 07:10:38 (1789297838) [ 1122.390788] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1124.521733] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1132.283920] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 07:10:49 (1789297849) [ 1142.508308] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 07:11:00 (1789297860) [ 1143.560612] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1153.593355] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1153.603215] Lustre: Skipped 2 previous similar messages [ 1155.750822] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 07:11:13 (1789297873) [ 1158.117257] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1158.122692] Lustre: Skipped 2 previous similar messages [ 1167.501709] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1167.515178] Lustre: Skipped 4 previous similar messages [ 1181.343182] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 07:11:39 (1789297899) [ 1183.861533] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1186.834829] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1194.833766] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 07:11:53 (1789297913) [ 1196.477752] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1196.480865] Lustre: Skipped 6 previous similar messages [ 1209.779928] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 07:12:07 (1789297927) [ 1224.165707] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1224.168641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1224.182959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1224.202748] Lustre: Skipped 1 previous similar message [ 1229.286715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1229.291188] Lustre: Skipped 1 previous similar message [ 1229.811262] Lustre: server umount lustre-OST0000 complete [ 1234.401555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1234.416105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1234.429972] Lustre: Skipped 2 previous similar messages [ 1239.521612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1239.527317] Lustre: Skipped 1 previous similar message [ 1244.643693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1244.646098] Lustre: Skipped 1 previous similar message [ 1248.226066] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1248.399056] Lustre: server umount lustre-MDT0000 complete [ 1249.765713] LustreError: 36050:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1249.789714] LustreError: 36050:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1252.490909] LustreError: 35014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789297972 with bad export cookie 12246810766341297906 [ 1252.505453] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1252.512253] LustreError: 35014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1253.114333] Lustre: server umount lustre-MDT0001 complete [ 1264.859490] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 1268.123434] Key type lgssc unregistered [ 1268.514084] LNet: 42890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1268.525638] LNetError: 42890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1268.547993] LNet: Removed LNI 192.168.204.157@tcp [ 1269.563777] Key type .llcrypt unregistered [ 1269.566884] Key type ._llcrypt unregistered [ 1290.065841] Key type ._llcrypt registered [ 1290.069860] Key type .llcrypt registered [ 1290.175689] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 1308.890463] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 1310.043125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1310.110204] alg: No test for adler32 (adler32-zlib) [ 1311.305578] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 1311.702034] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 1313.456636] Key type lgssc registered [ 1314.642282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1350.571651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1350.615487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1351.908474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1351.946284] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1352.037593] Lustre: lustre-MDT0000: new disk, initializing [ 1352.124563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1352.141746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1356.168450] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1366.339457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1366.494148] Lustre: 46052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1366.558241] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1366.563029] Lustre: Skipped 1 previous similar message [ 1366.634291] Lustre: lustre-MDT0001: new disk, initializing [ 1366.779242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1366.832856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1366.856886] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1371.476590] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1381.348287] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1387.594866] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1394.076126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1394.316951] Lustre: lustre-OST0000: new disk, initializing [ 1394.320757] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1394.326581] Lustre: 48016:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1394.407913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1400.895894] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1401.994651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1402.016177] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1402.159926] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1409.730913] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1414.395718] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1414.647317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1419.376200] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1419.574370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1422.822455] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1422.829497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1422.833715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1422.838207] Lustre: Skipped 1 previous similar message [ 1422.843630] LustreError: Skipped 1 previous similar message [ 1427.629910] Lustre: server umount lustre-OST0000 complete [ 1433.059451] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1433.066153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1433.074628] Lustre: Skipped 1 previous similar message [ 1433.087227] Lustre: Skipped 2 previous similar messages [ 1437.707453] Lustre: server umount lustre-MDT0000 complete [ 1441.144141] LustreError: 49356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789298161 with bad export cookie 251605175692022174 [ 1441.152548] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1441.155582] LustreError: 49356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1441.415568] Lustre: server umount lustre-MDT0001 complete [ 1450.298401] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 07:16:08 (1789298168) [ 1458.887556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1459.530038] LustreError: 50475:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1459.546342] LustreError: 50475:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1459.663197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1464.802879] LustreError: 50476:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1465.338727] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1469.922231] LustreError: 50475:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1474.978168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1475.045174] LustreError: 50476:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1475.332891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1479.285076] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1485.985881] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1491.403289] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1497.684855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1498.004286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1503.822741] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1510.618759] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1515.221741] LustreError: 53003:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1516.064331] Lustre: Modifying parameter general.timeout=1 in log params [ 1517.506534] Lustre: Modifying parameter general.timeout=3 in log params [ 1517.511224] Lustre: Skipped 1 previous similar message [ 1519.691970] Lustre: Modifying parameter general.timeout=6 in log params [ 1519.696575] Lustre: Skipped 2 previous similar messages [ 1523.856235] Lustre: Modifying parameter general.timeout=11 in log params [ 1523.862522] Lustre: Skipped 4 previous similar messages [ 1532.212418] Lustre: Modifying parameter general.timeout=22 in log params [ 1532.215180] Lustre: Skipped 10 previous similar messages [ 1548.520683] Lustre: Modifying parameter general.timeout=42 in log params [ 1548.525112] Lustre: Skipped 19 previous similar messages [ 1580.774422] Lustre: Modifying parameter general.timeout=77 in log params [ 1580.778846] Lustre: Skipped 34 previous similar messages [ 1637.284960] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 07:19:15 (1789298355) [ 1640.061336] Lustre: 59950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1641.737586] Lustre: 60048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1643.759308] Lustre: 60146:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1644.881773] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 in log params [ 1644.888503] Lustre: Skipped 57 previous similar messages [ 1647.931827] Lustre: 60359:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1647.943978] Lustre: 60359:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1677.208946] Lustre: 61832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1677.218854] Lustre: 61832:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1677.232758] LustreError: 61832:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1681.508459] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 07:19:59 (1789298399) [ 1695.723101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1695.728163] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1695.741326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1695.749569] Lustre: Skipped 1 previous similar message [ 1695.762285] Lustre: Skipped 2 previous similar messages [ 1700.543286] Lustre: server umount lustre-MDT0000 complete [ 1700.846316] LustreError: 50475:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1700.859407] LustreError: 50475:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1704.217349] LustreError: 50458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789298424 with bad export cookie 251605175692023056 [ 1704.217906] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1704.227941] LustreError: 50458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1704.501563] Lustre: server umount lustre-MDT0001 complete [ 1718.502432] Lustre: server umount lustre-OST0000 complete [ 1749.895477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1750.203751] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1750.509643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1756.147188] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1765.381195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1765.507445] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1765.531705] Lustre: 64366:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1770.982474] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1779.872520] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1786.684853] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1794.142724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1794.376057] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1794.674537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1794.680293] Lustre: Skipped 1 previous similar message [ 1801.888117] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1809.089577] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1810.850131] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1810.866505] Lustre: Skipped 18 previous similar messages [ 1810.944158] Lustre: 66931:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1823.716172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1823.718120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1823.727385] LustreError: Skipped 1 previous similar message [ 1823.733627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1823.747930] Lustre: Skipped 2 previous similar messages [ 1823.799734] Lustre: Skipped 1 previous similar message [ 1825.129817] Lustre: server umount lustre-OST0000 complete [ 1832.431191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1832.438337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1832.453201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1836.250486] Lustre: server umount lustre-MDT0000 complete [ 1839.089661] LustreError: 64372:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1839.108478] LustreError: 64372:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1839.518465] LustreError: 64357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789298559 with bad export cookie 251605175692026101 [ 1839.525986] LustreError: 64357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1839.527031] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1839.926165] Lustre: server umount lustre-MDT0001 complete [ 1851.868673] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 1854.650329] Key type lgssc unregistered [ 1855.008673] LNet: 68137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.019495] LNetError: 68137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1855.035596] LNet: Removed LNI 192.168.204.157@tcp [ 1855.786274] Key type .llcrypt unregistered [ 1855.788970] Key type ._llcrypt unregistered [ 1880.711264] Key type ._llcrypt registered [ 1880.714117] Key type .llcrypt registered [ 1880.817455] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 1882.058176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1882.105369] alg: No test for adler32 (adler32-zlib) [ 1883.158101] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 1883.353829] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 1885.000309] Key type lgssc registered [ 1886.412983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1894.117020] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1894.146519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1895.749415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1901.089663] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1905.134772] LustreError: 69549:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1905.154813] LustreError: 69549:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1910.072352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1910.266953] LustreError: 69548:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1910.839716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1915.650069] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1922.626032] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1927.524361] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1934.084917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1934.434193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1941.039860] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1947.989973] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1955.329203] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1960.934250] Lustre: server umount lustre-MDT0000 complete [ 1961.445157] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1961.449850] LustreError: 69543:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1961.452043] Lustre: Skipped 1 previous similar message [ 1961.468508] LustreError: 69543:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1964.064223] LustreError: 69530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789298684 with bad export cookie 10999651188542523533 [ 1964.072679] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1964.076510] LustreError: 69530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1964.089894] LustreError: 69540:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 1966.333184] Lustre: server umount lustre-MDT0001 complete [ 1992.871563] Lustre: server umount lustre-OST0000 complete [ 2022.175176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2022.511597] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2022.552242] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2022.931531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2028.532546] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2037.726979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2037.896115] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2037.920812] Lustre: 74418:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2038.389619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2043.023970] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2051.276711] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2057.613990] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2064.822325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2065.039531] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2065.364073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2071.544532] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2079.247639] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2080.928866] Lustre: 76993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2082.832149] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2089.955322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2089.959783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2089.965042] Lustre: Skipped 1 previous similar message [ 2089.973601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2090.469416] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2093.323748] Lustre: server umount lustre-OST0000 complete [ 2099.681400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2099.689973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2099.699251] Lustre: Skipped 1 previous similar message [ 2099.705907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2099.709584] Lustre: Skipped 2 previous similar messages [ 2100.706088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2103.225898] Lustre: server umount lustre-MDT0000 complete [ 2105.828762] LustreError: 74430:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.852416] LustreError: 74430:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2107.161175] LustreError: 76995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789298827 with bad export cookie 10999651188542524345 [ 2107.170215] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2107.175040] LustreError: 76995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2107.646313] Lustre: server umount lustre-MDT0001 complete [ 2120.552727] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 2123.752575] Key type lgssc unregistered [ 2124.036767] LNet: 78210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2124.042695] LNetError: 78210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2124.065766] LNet: Removed LNI 192.168.204.157@tcp [ 2125.079347] Key type .llcrypt unregistered [ 2125.084251] Key type ._llcrypt unregistered [ 2156.987930] Key type ._llcrypt registered [ 2156.992373] Key type .llcrypt registered [ 2157.162145] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2158.710573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2158.751907] alg: No test for adler32 (adler32-zlib) [ 2159.867698] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 2160.163810] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 2161.880241] Key type lgssc registered [ 2163.084582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2172.615695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2172.657239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2174.453123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2179.728421] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2184.673892] LustreError: 79618:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2184.706669] LustreError: 79618:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2189.798409] LustreError: 79619:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2191.545513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2192.560933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2197.924670] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2207.179483] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2214.804929] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2223.003915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2223.490838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2230.261860] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2238.792351] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2250.201562] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2251.554712] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2252.727963] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2254.879922] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2254.885878] Lustre: Skipped 1 previous similar message [ 2259.385236] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2259.389839] Lustre: Skipped 3 previous similar messages [ 2267.916937] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2267.921409] Lustre: Skipped 6 previous similar messages [ 2284.353161] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2284.357334] Lustre: Skipped 12 previous similar messages [ 2317.235685] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2317.240906] Lustre: Skipped 30 previous similar messages [ 2372.528073] Lustre: 87734:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2375.933403] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 07:31:33 (1789299093) [ 2378.440649] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2380.520717] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 07:31:38 (1789299098) [ 2386.406149] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2386.415787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2386.431460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2388.452061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2388.454634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2391.457147] Lustre: server umount lustre-OST0000 complete [ 2397.153820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2397.170545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2397.191305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2397.199965] Lustre: Skipped 1 previous similar message [ 2402.789435] Lustre: server umount lustre-MDT0000 complete [ 2403.812097] LustreError: 80351:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.839953] LustreError: 80351:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2407.482737] LustreError: 79598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789299127 with bad export cookie 1712201373485699137 [ 2407.503655] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2407.503841] LustreError: 79598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2408.204348] Lustre: server umount lustre-MDT0001 complete [ 2421.184818] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 2424.624991] Key type lgssc unregistered [ 2425.025328] LNet: 88982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2425.036367] LNetError: 88982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2425.054575] LNet: Removed LNI 192.168.204.157@tcp [ 2425.893486] Key type .llcrypt unregistered [ 2425.896091] Key type ._llcrypt unregistered [ 2430.965096] Key type ._llcrypt registered [ 2430.969582] Key type .llcrypt registered [ 2431.176325] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2436.540842] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules [ 2438.113936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2438.156931] alg: No test for adler32 (adler32-zlib) [ 2439.284406] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 2439.518897] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 2441.224218] Key type lgssc registered [ 2443.162480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2444.739514] LustreError: 90217:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2465.675800] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2468.073579] LustreError: 90217:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2473.970583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2473.988618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2475.466592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2479.396382] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2485.741606] LustreError: 91244:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2485.794649] LustreError: 91244:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2490.865071] LustreError: 91263:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2491.085588] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 07:33:29 (1789299209) [ 2495.974904] LustreError: 91244:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2496.171210] LustreError: 91277:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2496.179258] LustreError: 91277:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2496.186365] LustreError: 91277:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2496.599579] Lustre: server umount lustre-MDT0000 complete [ 2513.585982] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 2516.404793] Key type lgssc unregistered [ 2516.759511] LNet: 92770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2516.768533] LNetError: 92770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2516.788803] LNet: Removed LNI 192.168.204.157@tcp [ 2517.592466] Key type .llcrypt unregistered [ 2517.602147] Key type ._llcrypt unregistered [ 2536.253623] Key type ._llcrypt registered [ 2536.257389] Key type .llcrypt registered [ 2536.419192] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2537.623965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2537.671949] alg: No test for adler32 (adler32-zlib) [ 2538.819369] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 2539.276045] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 2540.969592] Key type lgssc registered [ 2542.243207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2550.829381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2550.861692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2552.591594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2557.450714] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2562.019755] LustreError: 93987:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2562.036086] LustreError: 93987:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2567.141777] LustreError: 93988:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.572586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2568.387896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2573.533972] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2581.279374] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2588.078943] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2596.619529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2597.496557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2603.676280] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2610.844401] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2667.504219] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 07:36:25 (1789299385) [ 2669.386796] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2671.486818] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 07:36:29 (1789299389) [ 2675.682853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2675.697197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2675.732881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2678.753573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2678.757947] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2678.770763] Lustre: Skipped 1 previous similar message [ 2681.385688] Lustre: server umount lustre-MDT0000 complete [ 2683.883479] LustreError: 95831:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2683.925526] LustreError: 95831:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2685.409322] LustreError: 93969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789299405 with bad export cookie 8673051816450346316 [ 2685.426862] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2685.981354] Lustre: server umount lustre-MDT0001 complete [ 2699.516523] Lustre: server umount lustre-OST0000 complete [ 2711.336506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2711.722140] LustreError: 98281:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2711.874418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2717.156950] LustreError: 98282:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2717.428349] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2726.373455] LustreError: 98281:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.383258] LustreError: 98281:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2726.657734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2727.465822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2731.930516] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2739.289691] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2745.085639] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2761.449892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2761.805429] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2761.828835] LustreError: 98269:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2761.863723] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2761.879475] LustreError: 100365:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2761.888276] LustreError: 100365:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2761.897451] LustreError: 100365:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2761.983415] Lustre: server umount lustre-OST0000 complete [ 2761.990485] LustreError: 100365:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2772.483698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2772.717616] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2772.726825] LustreError: 98270:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2772.748852] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2772.755925] LustreError: 100829:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2772.765845] LustreError: 100829:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2772.772512] LustreError: 100829:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2772.916714] Lustre: server umount lustre-OST0000 complete [ 2772.928707] LustreError: 100829:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2783.794445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2784.038697] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2784.050875] Lustre: Found index 0 for lustre-OST0000, updating log [ 2784.059697] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2784.177133] Lustre: lustre-OST0000: new disk, initializing [ 2784.186736] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2784.196470] Lustre: 101364:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2784.681353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2792.358735] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2796.238887] Lustre: Failing over lustre-OST0000 [ 2796.522909] Lustre: server umount lustre-OST0000 complete [ 2802.657912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2802.659057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2802.680649] Lustre: Skipped 1 previous similar message [ 2802.685328] Lustre: Skipped 2 previous similar messages [ 2807.779987] LustreError: 98282:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2807.803377] LustreError: 98282:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2807.871768] Lustre: server umount lustre-MDT0000 complete [ 2812.679784] LustreError: 98263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789299532 with bad export cookie 8673051816450349053 [ 2812.691498] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2812.693873] LustreError: 98263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2812.973787] Lustre: server umount lustre-MDT0001 complete [ 2823.200908] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 07:39:01 (1789299541) [ 2841.803976] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2852.307460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2852.711625] LustreError: 103713:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2852.810529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2857.993700] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2867.043974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.172521] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2872.964698] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2876.076627] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2882.760475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2889.985990] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2899.025487] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2899.151660] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2899.233548] Lustre: lustre-OST0001: new disk, initializing [ 2899.237707] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2899.244662] Lustre: 106030:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2899.524369] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2899.530625] Lustre: Skipped 2 previous similar messages [ 2906.278516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2906.291524] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2906.458563] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2906.537917] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2914.941668] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2926.306124] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2932.197954] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2932.210407] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2932.222475] Lustre: Skipped 1 previous similar message [ 2932.240739] Lustre: Skipped 2 previous similar messages [ 2940.454051] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.57@tcp (stopping) [ 2940.458408] Lustre: Skipped 4 previous similar messages [ 2944.992445] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2945.187581] Lustre: server umount lustre-MDT0001 complete [ 2945.557664] LustreError: 103710:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2945.573962] LustreError: 103710:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2955.738100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2955.948666] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2955.963270] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2956.049676] Lustre: 103703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2957.976122] LustreError: 107592:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2957.989084] LustreError: 107592:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2958.004064] LustreError: 103705:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2958.197610] LustreError: 107610:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.157@tcp: cfg command failed: rc = -17 [ 2958.204783] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2958.204783] [ 2961.905044] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2962.149104] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2976.000255] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 07:41:33 (1789299693) [ 2995.962662] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3014.210883] Lustre: 109648:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3039.161920] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3042.943950] Lustre: 110783:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3130.341153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3130.356200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3130.379874] Lustre: Skipped 1 previous similar message [ 3130.385931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3130.391135] Lustre: Skipped 3 previous similar messages [ 3135.794152] Lustre: server umount lustre-MDT0000 complete [ 3135.990547] LustreError: 107531:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3136.010336] LustreError: 107531:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 3140.002554] LustreError: 103695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789299860 with bad export cookie 8673051816450350054 [ 3140.007133] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3140.016055] LustreError: 103695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3140.344475] Lustre: server umount lustre-MDT0001 complete [ 3154.442839] Lustre: server umount lustre-OST0000 complete [ 3157.477887] Lustre: 93296:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789299861/real 1789299861] req@ffff9ae9ff25a680 x1876216260172288/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789299877 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3157.505509] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3157.517650] Lustre: Skipped 3 previous similar messages [ 3158.944603] Lustre: server umount lustre-OST0001 complete [ 3165.372179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3176.866698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3186.761824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3198.186595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3226.700203] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3241.810537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3241.834138] Lustre: lustre-MDT0000: reset Object Index mappings [ 3242.216954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3242.219774] Lustre: Skipped 1 previous similar message [ 3248.290426] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3258.724917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3258.755321] Lustre: lustre-MDT0001: reset Object Index mappings [ 3265.102268] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3268.627522] Lustre: 115908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3276.180855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3284.278449] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3284.410319] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 3287.489183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3293.722612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3299.820493] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3299.853384] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3301.817284] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3310.508290] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3314.466645] Lustre: 117785:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3370.215551] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 07:48:07 (1789300087) [ 3397.100203] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3397.103767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3397.108527] Lustre: Skipped 3 previous similar messages [ 3397.125081] Lustre: Skipped 6 previous similar messages [ 3401.006726] Lustre: server umount lustre-MDT0000 complete [ 3402.218897] LustreError: 116147:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3402.260581] LustreError: 116147:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 3404.916633] LustreError: 114741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789300124 with bad export cookie 8673051816450396926 [ 3404.930000] LustreError: 114741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3404.930837] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3411.724351] Lustre: server umount lustre-MDT0001 complete [ 3430.368136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3430.509532] Lustre: server umount lustre-OST0000 complete [ 3449.313437] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3449.529973] Lustre: server umount lustre-OST0001 complete [ 3457.802683] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 3470.117539] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3513.497912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3513.865626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3513.915942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3514.020452] Lustre: lustre-MDT0000: new disk, initializing [ 3514.076867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3514.084547] Lustre: Skipped 3 previous similar messages [ 3514.105893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3519.985644] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3527.682848] Lustre: server umount lustre-MDT0000 complete [ 3539.063491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3539.530033] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3539.565577] Lustre: 122062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3539.591278] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3545.209782] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3550.705891] Lustre: server umount lustre-MDT0000 complete [ 3560.371618] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 07:51:18 (1789300278) [ 3561.816910] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3564.043993] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 07:51:21 (1789300281) [ 3584.927670] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3598.563403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3604.771850] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3615.322149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3615.476961] Lustre: 124116:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3615.580884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3615.585221] Lustre: Skipped 1 previous similar message [ 3615.654568] Lustre: lustre-MDT0001: new disk, initializing [ 3615.837920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3615.865617] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3621.769616] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3627.228316] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3634.876835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3635.207959] Lustre: lustre-OST0000: new disk, initializing [ 3635.221268] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3635.232428] Lustre: 125695:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3636.570856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3636.594669] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3636.763287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3643.325796] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3656.047500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3656.311467] Lustre: lustre-OST0001: new disk, initializing [ 3656.315619] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3656.325562] Lustre: 126565:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3658.302356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3658.316484] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3658.489108] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3663.904377] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3676.875505] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3682.214974] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4149.310215] Lustre: 133006:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4149.322838] Lustre: 133006:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4149.332033] Lustre: 133006:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4149.337559] Lustre: 133006:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4149.342296] Lustre: 133006:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4149.353950] Lustre: 133006:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4487.651408] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 08:06:44 (1789301204) [ 4506.600396] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4506.608757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4506.641353] Lustre: Skipped 5 previous similar messages [ 4506.644757] Lustre: Skipped 6 previous similar messages [ 4516.833850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4516.836979] Lustre: Skipped 4 previous similar messages [ 4518.368487] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4518.684690] Lustre: server umount lustre-MDT0000 complete [ 4521.958867] LustreError: 127362:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4521.983296] LustreError: 127362:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4522.861919] LustreError: 133037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789301242 with bad export cookie 8673051816450466590 [ 4522.876812] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4522.885991] LustreError: 133037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4527.077647] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4537.312271] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4537.328208] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4537.338578] Lustre: Skipped 8 previous similar messages [ 4537.898058] Lustre: server umount lustre-MDT0001 complete [ 4547.685832] Lustre: server umount lustre-OST0000 complete [ 4557.591358] Lustre: server umount lustre-OST0001 complete [ 4565.640433] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 4578.385708] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 4622.004641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4622.255699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4622.296449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4622.372671] Lustre: lustre-MDT0000: new disk, initializing [ 4622.472508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4622.479434] Lustre: Skipped 5 previous similar messages [ 4622.492904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4627.807720] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4639.540235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4639.831298] Lustre: 136068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4639.930382] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4639.934915] Lustre: Skipped 1 previous similar message [ 4640.040608] Lustre: lustre-MDT0001: new disk, initializing [ 4640.245213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4640.253494] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4644.950852] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4655.696337] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4664.809927] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4673.820781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4674.641611] Lustre: lustre-OST0000: new disk, initializing [ 4674.647333] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4674.661907] Lustre: 138038:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4674.826072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4674.836387] Lustre: Skipped 1 previous similar message [ 4675.993309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4676.011609] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4676.178275] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4684.097654] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4694.025234] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4704.601745] Lustre: lustre-MDD0000: changelog on [ 4708.159588] Lustre: lustre-MDD0001: changelog on [ 6880.960254] Lustre: lustre-MDD0001: changelog off [ 6882.777130] Lustre: 138298:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6883.631535] Lustre: lustre-MDD0000: changelog off [ 6893.027874] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6893.032682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6893.037441] LustreError: Skipped 1 previous similar message [ 6893.038314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6893.062873] Lustre: Skipped 2 previous similar messages [ 6898.148334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6898.154700] Lustre: Skipped 1 previous similar message [ 6903.269086] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6903.442977] Lustre: server umount lustre-OST0000 complete [ 6908.387195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6908.393088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6908.408935] Lustre: Skipped 1 previous similar message [ 6908.416986] Lustre: Skipped 3 previous similar messages [ 6914.697552] Lustre: server umount lustre-MDT0000 complete [ 6918.626228] LustreError: 146193:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6918.654152] LustreError: 146193:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 6919.750328] LustreError: 136062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303639 with bad export cookie 8673051816451291722 [ 6919.773117] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6919.773388] LustreError: 136062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6920.584231] Lustre: server umount lustre-MDT0001 complete [ 6935.591684] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 6940.318582] Key type lgssc unregistered [ 6940.854046] LNet: 147219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6940.866372] LNetError: 147219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6940.895151] LNet: Removed LNI 192.168.204.157@tcp [ 6942.415212] Key type .llcrypt unregistered [ 6942.417196] Key type ._llcrypt unregistered [ 6967.591313] Key type ._llcrypt registered [ 6967.594297] Key type .llcrypt registered [ 6967.707153] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 6983.763220] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 6984.681536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6984.704761] alg: No test for adler32 (adler32-zlib) [ 6985.770583] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 6986.020526] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 6987.729632] Key type lgssc registered [ 6989.194623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7027.433638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7027.474253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7028.802954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7028.855902] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7028.930564] Lustre: lustre-MDT0000: new disk, initializing [ 7029.035080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7029.058351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7033.747087] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7046.798938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7047.074784] Lustre: 150421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7047.217353] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7047.227534] Lustre: Skipped 1 previous similar message [ 7047.416857] Lustre: lustre-MDT0001: new disk, initializing [ 7047.622051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7047.693558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7047.724494] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7053.346150] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7064.444083] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7071.189309] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7078.797768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7079.175701] Lustre: lustre-OST0000: new disk, initializing [ 7079.180131] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7079.188287] Lustre: 152392:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7079.349250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7087.043992] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7089.222569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7089.228505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7089.336983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7097.396228] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7105.507208] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_module obdecho/obdecho [ 7107.854357] LustreError: 153396:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7107.864642] LustreError: 153396:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7109.823946] LustreError: 153494:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7124.449639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7124.457873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7124.472926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7125.483742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7125.484117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7125.502997] Lustre: Skipped 1 previous similar message [ 7125.519761] Lustre: Skipped 2 previous similar messages [ 7128.911150] Lustre: server umount lustre-MDT0000 complete [ 7132.958925] LustreError: 150415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789303852 with bad export cookie 3152312057868072340 [ 7132.965379] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7132.980080] LustreError: 150415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7133.478554] Lustre: server umount lustre-MDT0001 complete [ 7148.905968] Lustre: server umount lustre-OST0000 complete [ 7173.408134] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 7185.105648] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 7225.493991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7225.736078] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7225.759786] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7225.863502] Lustre: lustre-MDT0000: new disk, initializing [ 7225.977099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7225.993704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7230.921247] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7242.792329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7242.958545] Lustre: 157212:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7243.007186] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7243.011310] Lustre: Skipped 1 previous similar message [ 7243.071276] Lustre: lustre-MDT0001: new disk, initializing [ 7243.184109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7243.230748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7243.248178] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7248.558581] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7258.486531] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7265.293528] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7273.103108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7273.464559] Lustre: lustre-OST0000: new disk, initializing [ 7273.481455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7273.498358] Lustre: 159178:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7273.655595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7280.537857] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7281.216974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7281.224229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7281.357591] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7291.104313] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7297.101812] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7297.362140] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7302.492868] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7302.710989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7307.239314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7307.251106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7307.264923] Lustre: Skipped 1 previous similar message [ 7307.267099] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7311.280280] Lustre: server umount lustre-OST0000 complete [ 7317.476460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7317.480403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7317.493385] Lustre: Skipped 1 previous similar message [ 7317.518702] Lustre: Skipped 2 previous similar messages [ 7321.478759] Lustre: server umount lustre-MDT0000 complete [ 7322.594042] LustreError: 157220:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7322.613504] LustreError: 157220:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 7325.550830] LustreError: 158006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304045 with bad export cookie 3152312057868073299 [ 7325.565707] LustreError: 158006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7325.569287] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7325.933174] Lustre: server umount lustre-MDT0001 complete [ 7330.252151] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 08:54:08 (1789304048) [ 7332.313305] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7334.937342] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 08:54:12 (1789304052) [ 7345.415204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7345.992164] LustreError: 161593:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7346.111689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7351.270740] LustreError: 161594:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7351.291243] LustreError: 161594:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7351.428817] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7356.394606] LustreError: 161593:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7361.404117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7361.506773] LustreError: 161594:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7367.436570] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7375.587132] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7382.040459] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7389.573436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7389.991123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7389.999862] Lustre: Skipped 1 previous similar message [ 7398.442322] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7408.347247] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7416.807143] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7416.817509] Lustre: Skipped 1 previous similar message [ 7416.820197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7421.310448] Lustre: server umount lustre-MDT0000 complete [ 7421.927320] LustreError: 161594:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7421.942610] LustreError: 161594:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7425.620820] LustreError: 161572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304145 with bad export cookie 3152312057868074174 [ 7425.627406] LustreError: 161572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7425.630803] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7427.043116] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7427.073158] Lustre: Skipped 1 previous similar message [ 7427.078641] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7427.082849] Lustre: Skipped 3 previous similar messages [ 7432.095516] Lustre: server umount lustre-MDT0001 complete [ 7439.654604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7454.004424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7454.297534] LustreError: 165438:0:(llog.c:1655:llog_backup()) MGC192.168.204.157@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7454.309225] Lustre: 165438:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.157@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7455.715298] LustreError: 165442:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7455.743916] LustreError: 165442:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7490.533687] LustreError: 165445:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7490.559155] LustreError: 165445:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 26 previous similar messages [ 7515.616591] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7515.626809] Lustre: 165438:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.157@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7556.599256] LustreError: 165464:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7556.612286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7556.628126] LustreError: 165464:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 44 previous similar messages [ 7561.700735] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7562.090407] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7572.445952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7573.213979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -11 [ 7573.224286] LustreError: Skipped 1 previous similar message [ 7578.611613] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7578.766094] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7587.663711] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7589.624159] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7596.300432] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7598.212929] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7620.294924] Lustre: server umount lustre-MDT0000 complete [ 7624.503371] LustreError: 161574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304344 with bad export cookie 3152312057868080635 [ 7624.516136] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7624.526942] LustreError: 161574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7624.681151] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7624.698398] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7624.701486] Lustre: Skipped 1 previous similar message [ 7624.711440] Lustre: Skipped 2 previous similar messages [ 7624.959763] Lustre: server umount lustre-MDT0001 complete [ 7630.220353] Lustre: server umount lustre-OST0000 complete [ 7655.401213] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 7667.420512] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 7713.051088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7713.475607] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7713.546603] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7713.703970] Lustre: lustre-MDT0000: new disk, initializing [ 7713.855052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7713.861220] Lustre: Skipped 1 previous similar message [ 7713.891598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7719.939122] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7733.051833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7733.272612] Lustre: 170545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7733.644145] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7739.946796] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7751.389848] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7758.560429] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7767.252186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7767.797914] Lustre: lustre-OST0000: new disk, initializing [ 7767.801744] Lustre: Skipped 1 previous similar message [ 7767.813315] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7767.821711] Lustre: Skipped 2 previous similar messages [ 7767.834340] Lustre: 172526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7775.726935] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7777.388913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7777.397762] Lustre: Skipped 1 previous similar message [ 7777.411974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7777.560313] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7786.176793] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7791.999799] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7792.288135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7797.210576] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7797.505370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7803.372291] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7803.382865] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7803.389159] LustreError: Skipped 1 previous similar message [ 7803.390928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7803.422038] Lustre: Skipped 1 previous similar message [ 7814.112194] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7814.425963] Lustre: server umount lustre-OST0000 complete [ 7824.708383] Lustre: server umount lustre-MDT0000 complete [ 7828.961862] LustreError: 170556:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7829.001833] LustreError: 170556:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 18 previous similar messages [ 7830.150412] LustreError: 170536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304550 with bad export cookie 3152312057868097463 [ 7830.152500] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7830.167941] LustreError: 170536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7830.712890] Lustre: server umount lustre-MDT0001 complete [ 7844.666436] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 09:02:41 (1789304561) [ 7853.990299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7860.291571] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7871.202620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7876.668332] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7885.367691] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7891.990310] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7899.910194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7907.828791] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 7916.300335] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7937.690717] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 09:04:15 (1789304655) [ 7942.627110] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7942.639182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7942.657704] Lustre: Skipped 2 previous similar messages [ 7942.661414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7942.666664] Lustre: Skipped 9 previous similar messages [ 7947.657964] Lustre: server umount lustre-OST0000 complete [ 7953.377626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7953.390299] LustreError: Skipped 1 previous similar message [ 7965.152142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7965.286958] Lustre: server umount lustre-MDT0000 complete [ 7968.483536] LustreError: 174922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304688 with bad export cookie 3152312057868098352 [ 7968.494591] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7968.502353] LustreError: 174922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7968.825687] Lustre: server umount lustre-MDT0001 complete [ 7979.400712] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 7982.669748] Key type lgssc unregistered [ 7982.993665] LNet: 178695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7983.000644] LNetError: 178695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7983.022649] LNet: Removed LNI 192.168.204.157@tcp [ 7983.970644] Key type .llcrypt unregistered [ 7983.975842] Key type ._llcrypt unregistered [ 8003.289423] Key type ._llcrypt registered [ 8003.291503] Key type .llcrypt registered [ 8003.405890] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 8004.260555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8004.423523] alg: No test for adler32 (adler32-zlib) [ 8005.567340] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 8005.807360] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 8007.536176] Key type lgssc registered [ 8008.574223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8016.549816] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8016.574272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8032.160608] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8037.280696] LustreError: 179935:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.157@tcp: failed processing log, type 4: rc = -110 [ 8042.400682] LustreError: 179935:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8042.435786] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8042.456339] LustreError: MGC192.168.204.157@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8042.478419] LustreError: 179935:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8042.492891] LustreError: 179935:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 8042.501762] LustreError: 179935:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 8042.562036] Lustre: server umount lustre-OST0000 complete [ 8042.567462] LustreError: 179935:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8052.312850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8053.868120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8058.453802] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8063.464549] LustreError: 180434:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8063.481214] LustreError: 180434:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8067.574805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8067.945785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8072.640967] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8080.174395] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8086.312660] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8093.233386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8093.441865] LustreError: 182251:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8093.459813] Lustre: 182251:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.157@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8093.621291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8099.981667] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8106.478053] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8109.027190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8109.046321] Lustre: Skipped 1 previous similar message [ 8109.047145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8109.074093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8112.164230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8112.176415] Lustre: Skipped 4 previous similar messages [ 8115.039843] Lustre: server umount lustre-MDT0000 complete [ 8117.219906] LustreError: 180434:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8117.233347] LustreError: 180434:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8118.116063] LustreError: 180415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304838 with bad export cookie 18396185472251080990 [ 8118.120569] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8118.129031] LustreError: 180415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8118.328560] LustreError: 181163:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002de39588 x1876221991349888/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8118.460644] Lustre: server umount lustre-MDT0001 complete [ 8131.275071] Lustre: server umount lustre-OST0000 complete [ 8139.715155] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 8150.011549] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 8185.239806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8185.463563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8185.493695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8185.550913] Lustre: lustre-MDT0000: new disk, initializing [ 8185.603828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8185.620463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8189.979408] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8199.118541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8199.243462] Lustre: 185773:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8199.283459] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8199.289753] Lustre: Skipped 1 previous similar message [ 8199.316500] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8199.368312] Lustre: lustre-MDT0001: new disk, initializing [ 8199.436152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8199.472543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8199.479513] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8203.651334] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8212.111554] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8217.581561] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8224.151632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8224.405439] Lustre: lustre-OST0000: new disk, initializing [ 8224.410988] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8224.419611] Lustre: 187740:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8224.492939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8230.029859] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8233.010023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8233.019231] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8233.110652] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8238.666810] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8242.979047] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8243.241512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8247.862886] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8248.082968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8253.416788] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8253.423670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8253.449709] Lustre: Skipped 1 previous similar message [ 8253.460502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8256.623259] Lustre: server umount lustre-OST0000 complete [ 8261.092335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8261.107052] LustreError: Skipped 1 previous similar message [ 8261.114111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8261.133281] Lustre: Skipped 1 previous similar message [ 8261.151059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8261.154691] Lustre: Skipped 1 previous similar message [ 8264.163796] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8267.292970] Lustre: server umount lustre-MDT0000 complete [ 8269.283770] LustreError: 185785:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8269.292144] LustreError: 185785:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8270.483481] LustreError: 186582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789304990 with bad export cookie 18396185472251081711 [ 8270.488477] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8270.494426] LustreError: 186582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8270.799918] Lustre: server umount lustre-MDT0001 complete [ 8278.205858] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 09:09:56 (1789304996) [ 8292.721417] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 8295.124323] Key type lgssc unregistered [ 8295.393039] LNet: 190683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8295.398991] LNetError: 190683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8296.424674] LNet: Removed LNI 192.168.204.157@tcp [ 8297.094878] Key type .llcrypt unregistered [ 8297.100430] Key type ._llcrypt unregistered [ 8313.303912] Key type ._llcrypt registered [ 8313.305850] Key type .llcrypt registered [ 8313.394558] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 8314.431391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8314.458445] alg: No test for adler32 (adler32-zlib) [ 8315.431459] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 8315.662591] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 8317.352373] Key type lgssc registered [ 8318.162966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8324.656754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8324.678407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8326.042267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8329.977663] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8336.354981] LustreError: 191899:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8336.386176] LustreError: 191899:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8337.712294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8337.987895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8341.768498] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8348.271774] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8353.303643] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8359.549237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8359.718040] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8359.721607] LustreError: 193760:0:(llog.c:1696:llog_backup()) MGC192.168.204.157@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8359.726805] Lustre: 193760:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.157@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8359.794852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8364.694973] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8371.061177] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8373.729712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8373.733381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8373.744879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8376.293104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8376.295337] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8378.907607] Lustre: server umount lustre-MDT0000 complete [ 8381.412899] LustreError: 191899:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8381.525899] LustreError: 194522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305101 with bad export cookie 13028527068969251316 [ 8381.529103] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8381.532088] LustreError: 194522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8381.775520] Lustre: server umount lustre-MDT0001 complete [ 8394.211481] Lustre: server umount lustre-OST0000 complete [ 8401.235281] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 8409.119340] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 8438.226923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8438.388908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8438.406933] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8438.463262] Lustre: lustre-MDT0000: new disk, initializing [ 8438.512566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8438.523172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8441.165995] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8448.932851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8449.014479] Lustre: 197285:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8449.052196] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8449.054517] Lustre: Skipped 1 previous similar message [ 8449.057191] LustreError: 198066:0:(llog.c:1696:llog_backup()) MGC192.168.204.157@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8449.065777] LustreError: 198066:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8449.070235] Lustre: 198066:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.157@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8449.086764] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8449.090615] Lustre: Skipped 1 previous similar message [ 8449.137876] Lustre: lustre-MDT0001: new disk, initializing [ 8449.220986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8449.250910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8449.258328] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8452.662948] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8460.635259] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8465.095388] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8470.652306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8470.885487] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8470.887511] Lustre: Skipped 1 previous similar message [ 8470.890490] LustreError: 199229:0:(llog.c:1696:llog_backup()) MGC192.168.204.157@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8470.898454] LustreError: 199229:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8470.903102] Lustre: 199229:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.157@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8470.909332] Lustre: 199229:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8470.925770] Lustre: lustre-OST0000: new disk, initializing [ 8470.929187] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8470.933972] Lustre: 199252:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8471.024912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8475.281684] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8478.749567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8478.757464] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8478.831034] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8481.968301] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8485.395797] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8485.595293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8488.823856] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8488.995931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8494.050104] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8494.050980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8494.063100] LustreError: Skipped 1 previous similar message [ 8494.065800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8494.065808] Lustre: Skipped 2 previous similar messages [ 8494.087514] Lustre: Skipped 2 previous similar messages [ 8496.694321] Lustre: server umount lustre-OST0000 complete [ 8500.707162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8500.718803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8500.737513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8500.741833] Lustre: Skipped 1 previous similar message [ 8505.887099] Lustre: server umount lustre-MDT0000 complete [ 8508.830511] LustreError: 197275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305228 with bad export cookie 13028527068969252037 [ 8508.841054] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8508.845377] LustreError: 197275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8509.124529] Lustre: server umount lustre-MDT0001 complete [ 8516.197236] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 09:13:54 (1789305234) [ 8528.604652] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 8536.414843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8536.820962] LustreError: 202377:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8536.829198] LustreError: 202377:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8536.861884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8540.226480] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8542.189742] LustreError: 202378:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8546.279816] LustreError: 202377:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8546.618272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8550.164315] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8552.508983] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8557.946726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8562.590700] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8568.130710] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8568.237648] Lustre: lustre-OST0001: new disk, initializing [ 8568.241518] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8568.248214] Lustre: 204681:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8572.745651] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8574.013627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8574.021239] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8574.086937] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8580.982270] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8585.407811] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8586.922669] Lustre: DEBUG MARKER: ADD OST3 [ 8590.575573] loop: module loaded [ 8590.633799] loop0: detected capacity change from 0 to 204800000 [ 8590.838412] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8590.847834] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8590.857678] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8595.574354] loop0: detected capacity change from 0 to 204800000 [ 8597.092436] Lustre: DEBUG MARKER: STOP OST3 [ 8599.936591] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8600.083826] Lustre: lustre-OST0003: new disk, initializing [ 8600.088021] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8600.094156] Lustre: 206620:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8600.176337] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8600.186633] Lustre: Skipped 3 previous similar messages [ 8601.276117] Lustre: Failing over lustre-OST0003 [ 8601.363425] Lustre: server umount lustre-OST0003 complete [ 8601.576348] LustreError: 205130:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8604.145680] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8611.319071] Lustre: DEBUG MARKER: START OST3 again [ 8616.170409] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8620.901597] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8621.560344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8621.565502] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8621.590446] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 8638.960899] Lustre: Failing over lustre-OST0003 [ 8639.013818] Lustre: server umount lustre-OST0003 complete [ 8642.036168] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8642.046667] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8642.060624] Lustre: Skipped 1 previous similar message [ 8642.066424] LustreError: 203862:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8642.078105] LustreError: 203862:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 8654.305773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8654.310649] LustreError: Skipped 1 previous similar message [ 8654.314590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8654.321771] Lustre: Skipped 1 previous similar message [ 8654.325933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8654.330520] Lustre: Skipped 2 previous similar messages [ 8660.577548] Lustre: server umount lustre-MDT0000 complete [ 8663.030372] LustreError: 205510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305383 with bad export cookie 13028527068969252919 [ 8663.033308] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8663.041986] LustreError: 205510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8667.620571] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8667.629923] Lustre: Skipped 5 previous similar messages [ 8677.344163] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8677.599351] Lustre: server umount lustre-MDT0001 complete [ 8680.077672] Lustre: server umount lustre-OST0000 complete [ 8682.438066] Lustre: server umount lustre-OST0001 complete [ 8687.130854] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 8693.211626] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 8722.568784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8722.767405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8722.786156] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8722.845893] Lustre: lustre-MDT0000: new disk, initializing [ 8722.918496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8722.927600] Lustre: Skipped 1 previous similar message [ 8722.943448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8726.733908] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8736.017082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8736.092749] Lustre: 212143:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8736.319982] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8739.194309] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8745.703423] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8749.853151] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8754.623711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8754.819934] Lustre: lustre-OST0000: new disk, initializing [ 8754.823429] Lustre: Skipped 1 previous similar message [ 8754.825663] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8754.829024] Lustre: Skipped 2 previous similar messages [ 8754.833220] Lustre: 214110:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8756.704279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8756.712915] Lustre: Skipped 1 previous similar message [ 8756.715370] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8756.784521] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8759.096668] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8765.598378] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8768.478534] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8768.617091] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8771.539693] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8771.673583] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8777.195206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8777.199350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8777.204072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8777.204081] Lustre: Skipped 4 previous similar messages [ 8777.206802] LustreError: Skipped 1 previous similar message [ 8777.227398] Lustre: Skipped 6 previous similar messages [ 8779.038082] Lustre: server umount lustre-OST0000 complete [ 8787.640153] Lustre: server umount lustre-MDT0000 complete [ 8789.945871] LustreError: 212947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305509 with bad export cookie 13028527068969255194 [ 8789.946246] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8789.957020] LustreError: 212947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8790.133263] Lustre: server umount lustre-MDT0001 complete [ 8792.496683] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 09:18:31 (1789305511) [ 8802.454795] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 8809.131463] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 8831.747430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8831.876644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8831.894129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8831.947503] Lustre: lustre-MDT0000: new disk, initializing [ 8832.027527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8835.098468] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8843.365574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8843.466389] Lustre: 218714:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8843.648337] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8847.107660] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8853.698978] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8857.509767] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8862.336267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8862.509487] Lustre: 220686:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8862.590387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8862.593791] Lustre: Skipped 4 previous similar messages [ 8864.388465] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8866.622712] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8873.589237] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8876.766435] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8876.909109] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8879.958229] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8880.116368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8884.712641] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8884.713865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8884.719399] LustreError: Skipped 1 previous similar message [ 8884.735775] Lustre: Skipped 2 previous similar messages [ 8884.740763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8884.747493] Lustre: Skipped 6 previous similar messages [ 8887.535502] Lustre: server umount lustre-OST0000 complete [ 8896.236198] Lustre: server umount lustre-MDT0000 complete [ 8898.821879] LustreError: 218707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305618 with bad export cookie 13028527068969256076 [ 8898.823789] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8898.830317] LustreError: 218707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8899.071388] Lustre: server umount lustre-MDT0001 complete [ 8905.099684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8905.386321] LustreError: 222900:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8905.401954] LustreError: 222900:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 16 previous similar messages [ 8908.160543] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8913.690279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8916.168733] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8920.154803] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8923.226603] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8927.289738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8931.167718] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 8935.775332] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8946.145705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8946.157040] LustreError: Skipped 1 previous similar message [ 8949.220554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8949.224571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8949.227045] Lustre: Skipped 3 previous similar messages [ 8949.238081] Lustre: Skipped 4 previous similar messages [ 8950.690438] Lustre: server umount lustre-OST0000 complete [ 8959.362295] Lustre: server umount lustre-MDT0000 complete [ 8961.800107] LustreError: 224743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305681 with bad export cookie 13028527068969256958 [ 8961.805421] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8961.808401] LustreError: 224743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8962.045786] Lustre: server umount lustre-MDT0001 complete [ 9069.930492] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 9071.611697] Key type lgssc unregistered [ 9071.777669] LNet: 226562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9071.782590] LNetError: 226562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9071.796030] LNet: Removed LNI 192.168.204.157@tcp [ 9072.254157] Key type .llcrypt unregistered [ 9072.256030] Key type ._llcrypt unregistered [ 9087.047155] Key type ._llcrypt registered [ 9087.048743] Key type .llcrypt registered [ 9087.127133] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 9096.564527] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 9097.169558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9097.196166] alg: No test for adler32 (adler32-zlib) [ 9098.135809] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 9098.299747] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 9099.936211] Key type lgssc registered [ 9100.641521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9122.326888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9122.340141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9123.485546] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9123.508873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9123.552912] Lustre: lustre-MDT0000: new disk, initializing [ 9123.597593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9123.612236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9126.456935] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9133.327049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9133.409322] Lustre: 229884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9133.461847] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9133.466155] Lustre: Skipped 1 previous similar message [ 9133.523443] Lustre: lustre-MDT0001: new disk, initializing [ 9133.599387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9133.625438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9133.629590] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9136.277110] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9142.174541] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9145.487108] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9149.740248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9149.889639] Lustre: lustre-OST0000: new disk, initializing [ 9149.892792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9149.897315] Lustre: 231849:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9149.960955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9153.087704] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9156.133772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9156.141581] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9156.218421] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9159.158951] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9161.682972] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9161.824387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9164.355421] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9164.493385] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9166.306253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9166.308587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9166.309725] LustreError: Skipped 1 previous similar message [ 9166.311686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9166.318491] Lustre: Skipped 1 previous similar message [ 9171.429262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9171.435254] Lustre: Skipped 2 previous similar messages [ 9171.782084] Lustre: server umount lustre-OST0000 complete [ 9174.499946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9174.505574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9174.514104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9176.548243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9176.551729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9181.665512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9181.671394] Lustre: Skipped 2 previous similar messages [ 9188.320107] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9188.431609] Lustre: server umount lustre-MDT0000 complete [ 9190.747329] LustreError: 230678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305910 with bad export cookie 3615987941111908154 [ 9190.749571] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9190.753048] LustreError: 230678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9190.955629] Lustre: server umount lustre-MDT0001 complete [ 9201.463382] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 9207.475328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9207.700127] LustreError: 234781:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9207.712900] LustreError: 234781:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9207.738774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9210.356653] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9212.900292] LustreError: 234782:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9216.051230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9216.295033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9218.960045] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9220.741407] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9224.981206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9225.175556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9228.920700] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9234.206810] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9234.324088] Lustre: lustre-OST0001: new disk, initializing [ 9234.328204] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9234.332676] Lustre: 237087:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9238.000548] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9239.077422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9239.084430] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9239.134413] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9246.363477] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9249.218830] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9256.051194] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 09:26:15 (1789305975) [ 9257.441116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9257.447799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9257.457387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9257.460064] Lustre: Skipped 2 previous similar messages [ 9259.311617] Lustre: server umount lustre-MDT0000 complete [ 9259.495640] LustreError: 234781:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9259.502284] LustreError: 234781:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9261.264973] LustreError: 237086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789305981 with bad export cookie 3615987941111909043 [ 9261.267461] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9261.271517] LustreError: 237086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9261.461441] Lustre: server umount lustre-MDT0001 complete [ 9273.875992] Lustre: server umount lustre-OST0000 complete [ 9285.952491] Lustre: server umount lustre-OST0001 complete [ 9289.582691] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 9295.032475] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 9313.640559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9313.749506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9313.764555] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9313.803822] Lustre: lustre-MDT0000: new disk, initializing [ 9313.833881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9313.837541] Lustre: Skipped 1 previous similar message [ 9313.845738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9315.682696] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9321.513784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9321.595447] Lustre: 241028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9321.770586] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9323.941489] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9329.405505] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9332.415061] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9336.056222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9336.217931] Lustre: lustre-OST0000: new disk, initializing [ 9336.220368] Lustre: Skipped 1 previous similar message [ 9336.223542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9336.226778] Lustre: Skipped 2 previous similar messages [ 9336.230817] Lustre: 242995:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9337.377275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9337.382442] Lustre: Skipped 1 previous similar message [ 9337.384808] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9337.467404] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9339.596351] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9344.661437] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9346.701325] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9346.791156] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9348.890537] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9349.012649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9352.675712] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9352.680450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9352.684025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9352.688059] Lustre: Skipped 4 previous similar messages [ 9352.696212] Lustre: Skipped 1 previous similar message [ 9356.215463] Lustre: server umount lustre-OST0000 complete [ 9362.916280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9362.917310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9362.927418] Lustre: Skipped 1 previous similar message [ 9364.217200] Lustre: server umount lustre-MDT0000 complete [ 9366.036727] LustreError: 242994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306086 with bad export cookie 3615987941111911178 [ 9366.042268] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9366.045233] LustreError: 242994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9366.188879] Lustre: server umount lustre-MDT0001 complete [ 9370.413663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9370.648616] LustreError: 245207:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9370.691251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9370.697738] Lustre: Skipped 2 previous similar messages [ 9372.778311] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9375.713938] LustreError: 245207:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9376.780047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9378.956893] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9382.805562] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9385.572149] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9389.164949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9392.494564] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9396.728545] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9544.161327] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9544.165845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9544.173836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9544.176939] Lustre: Skipped 2 previous similar messages [ 9547.494714] Lustre: server umount lustre-OST0000 complete [ 9555.364334] Lustre: server umount lustre-MDT0000 complete [ 9555.425170] LustreError: 247708:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9555.433115] LustreError: 247708:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9557.060552] LustreError: 245189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306277 with bad export cookie 3615987941111912060 [ 9557.061186] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9557.065649] LustreError: 245189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9557.227145] Lustre: server umount lustre-MDT0001 complete [ 9663.825709] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 9665.359389] Key type lgssc unregistered [ 9665.541872] LNet: 248812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9665.545970] LNetError: 248812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9665.553781] LNet: Removed LNI 192.168.204.157@tcp [ 9666.042190] Key type .llcrypt unregistered [ 9666.043822] Key type ._llcrypt unregistered [ 9678.227841] Key type ._llcrypt registered [ 9678.229636] Key type .llcrypt registered [ 9678.277818] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 9686.099542] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 9686.543080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9686.645567] alg: No test for adler32 (adler32-zlib) [ 9687.534824] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 9687.654745] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 9689.256202] Key type lgssc registered [ 9689.766879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9710.313703] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 9714.597038] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9714.604757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9715.704739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9715.718677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9715.758388] Lustre: lustre-MDT0000: new disk, initializing [ 9715.785202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9715.792527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9717.494849] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9722.855776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9722.914477] Lustre: 252850:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9722.938965] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9722.941380] Lustre: Skipped 1 previous similar message [ 9722.981144] Lustre: lustre-MDT0001: new disk, initializing [ 9723.027458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9723.047348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9723.052818] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9724.827205] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9727.458652] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9730.194334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9730.320603] Lustre: lustre-OST0000: new disk, initializing [ 9730.322806] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9730.326367] Lustre: 254483:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9730.370087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9732.695097] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9732.808981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9732.814196] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9732.864767] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9738.046947] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9738.125402] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up) [ 9738.135162] Lustre: lustre-OST0001: new disk, initializing [ 9738.137458] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9738.141189] Lustre: 255359:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9738.179812] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9740.773523] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9743.356265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9743.360663] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9743.375710] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000400 [ 9747.207326] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9753.390819] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9759.201535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9759.207080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9759.214727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9760.740072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9760.741860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9760.747665] Lustre: Skipped 2 previous similar messages [ 9760.753340] Lustre: Skipped 2 previous similar messages [ 9763.459565] Lustre: server umount lustre-MDT0000 complete [ 9765.035187] LustreError: 252843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306485 with bad export cookie 2383986779212811968 [ 9765.039173] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9765.041313] LustreError: 252843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9765.175802] Lustre: server umount lustre-MDT0001 complete [ 9777.050810] Lustre: server umount lustre-OST0000 complete [ 9788.090733] Lustre: server umount lustre-OST0001 complete [ 9794.204379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9798.220456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9802.156478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9806.968158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9816.325164] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 9821.245503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9821.255561] Lustre: lustre-MDT0000: reset Object Index mappings [ 9821.493112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9823.524592] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9826.787000] LustreError: 260364:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9826.793717] LustreError: 260364:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9827.753659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9827.762511] Lustre: lustre-MDT0001: reset Object Index mappings [ 9829.787853] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9831.044925] Lustre: 261504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9834.015042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9834.182867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9834.186132] Lustre: Skipped 1 previous similar message [ 9836.563040] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9837.287571] LustreError: 261859:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9840.309402] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9843.067223] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9846.762895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9846.762894] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 9852.152849] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9853.712311] Lustre: 263374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9856.996702] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9856.998957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9857.005879] Lustre: Skipped 2 previous similar messages [ 9857.008237] Lustre: Skipped 2 previous similar messages [ 9861.520801] Lustre: server umount lustre-MDT0000 complete [ 9862.113033] LustreError: 260363:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9862.120532] LustreError: 260363:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9863.089103] LustreError: 260344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306583 with bad export cookie 2383986779212818744 [ 9863.092626] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9863.094352] LustreError: 260344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9863.232783] Lustre: server umount lustre-MDT0001 complete [ 9874.437938] Lustre: server umount lustre-OST0000 complete [ 9885.235703] Lustre: server umount lustre-OST0001 complete [ 9893.437152] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 09:36:52 (1789306612) [ 9900.512765] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 9904.909393] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 9919.692428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9919.789176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9919.806837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9919.848791] Lustre: lustre-MDT0000: new disk, initializing [ 9919.885820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9919.888892] Lustre: Skipped 1 previous similar message [ 9919.897514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9921.633399] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9926.393239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9926.441506] Lustre: 267618:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9926.569910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9928.326676] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9932.771843] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9935.070513] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9937.923113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9938.050162] Lustre: lustre-OST0000: new disk, initializing [ 9938.052022] Lustre: Skipped 1 previous similar message [ 9938.054365] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9938.056863] Lustre: Skipped 2 previous similar messages [ 9938.059710] Lustre: 269583:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9940.103526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9940.107535] Lustre: Skipped 1 previous similar message [ 9940.110343] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9940.159118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9940.689871] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9945.396881] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9947.293281] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9947.373626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9948.983031] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9949.072707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9950.690659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9950.691099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9950.693752] LustreError: Skipped 1 previous similar message [ 9950.699045] Lustre: Skipped 1 previous similar message [ 9950.701581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9950.704923] Lustre: Skipped 1 previous similar message [ 9956.014951] Lustre: server umount lustre-OST0000 complete [ 9960.930618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9960.931923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9960.935643] Lustre: Skipped 1 previous similar message [ 9960.940851] Lustre: Skipped 3 previous similar messages [ 9963.746940] Lustre: server umount lustre-MDT0000 complete [ 9965.248309] LustreError: 267611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306685 with bad export cookie 2383986779212820816 [ 9965.251180] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9965.253032] LustreError: 267611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9965.383219] Lustre: server umount lustre-MDT0001 complete [ 9972.204680] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 9976.056840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9976.189735] LustreError: 272514:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9976.202427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9976.204437] Lustre: Skipped 2 previous similar messages [ 9977.787861] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9981.409863] LustreError: 272515:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9981.579696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9983.497777] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9984.752472] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9987.357214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9989.783665] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 9993.206892] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9993.284600] Lustre: lustre-OST0001: new disk, initializing [ 9993.287251] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9993.291240] Lustre: 274821:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9994.702516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9994.707597] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9994.750656] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9995.680968] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10000.779599] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10002.273109] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10004.495978] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10004.504143] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10012.131184] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10012.134017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10012.135061] Lustre: Skipped 3 previous similar messages [10012.136661] Lustre: Skipped 2 previous similar messages [10014.063357] Lustre: server umount lustre-MDT0000 complete [10015.527436] LustreError: 273249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306735 with bad export cookie 2383986779212821691 [10015.531757] LustreError: 273249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10015.532904] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10015.651027] Lustre: server umount lustre-MDT0001 complete [10027.470948] Lustre: server umount lustre-OST0000 complete [10039.347373] Lustre: server umount lustre-OST0001 complete [10045.947125] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [10050.394262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10050.564301] LustreError: 278156:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10050.590031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10050.593847] Lustre: Skipped 3 previous similar messages [10052.211046] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10055.683651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10057.556946] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10058.734104] Lustre: 279297:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10061.288477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10063.509889] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10066.504164] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10068.743273] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10075.117670] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10075.128519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10082.553093] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10083.828995] Lustre: 281196:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10085.808901] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10085.818836] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10090.425191] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 09:40:09 (1789306809) [10102.241280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10102.245266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10102.250722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10108.127996] Lustre: server umount lustre-MDT0000 complete [10109.587390] LustreError: 281198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306829 with bad export cookie 2383986779212828068 [10109.589521] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10109.592833] LustreError: 281198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10110.945485] LustreError: 279672:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10110.951357] LustreError: 279672:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10115.786449] Lustre: server umount lustre-MDT0001 complete [10123.455521] Lustre: server umount lustre-OST0000 complete [10139.104130] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10139.174542] Lustre: server umount lustre-OST0001 complete [10141.496242] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [10145.177915] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [10161.574506] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [10165.319492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10165.407095] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10165.418059] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10165.454105] Lustre: lustre-MDT0000: new disk, initializing [10165.479965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10166.959712] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10171.578809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10171.626805] Lustre: 285715:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10171.748132] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10173.373812] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10178.492116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10178.618688] Lustre: 287349:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10178.660470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10178.663847] Lustre: Skipped 5 previous similar messages [10179.517711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10180.892635] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10185.158655] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10187.110455] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10187.267775] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10191.769158] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10197.524984] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 09:41:56 (1789306916) [10202.593753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10202.595143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10202.597854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10202.597861] Lustre: Skipped 8 previous similar messages [10202.607520] Lustre: Skipped 6 previous similar messages [10204.450179] Lustre: server umount lustre-MDT0000 complete [10205.814282] LustreError: 286511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789306925 with bad export cookie 2383986779212836020 [10205.818087] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10205.818867] LustreError: 286511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10205.941369] Lustre: server umount lustre-MDT0001 complete [10217.928942] Lustre: server umount lustre-OST0000 complete [10229.674229] Lustre: server umount lustre-OST0001 complete [10232.854260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10233.026381] LustreError: 290459:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10233.035501] LustreError: 290459:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10234.623796] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10238.076908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10239.740743] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10242.411719] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10244.373253] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10246.773711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10248.968514] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10251.621906] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10268.230112] loop1: detected capacity change from 0 to 204800000 [10268.318108] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10268.322349] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10268.326501] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10270.059633] loop1: detected capacity change from 0 to 204800000 [10272.036524] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10272.070456] LustreError: 290447:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs22207: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10272.078177] LustreError: 290447:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs22207 [10272.082251] LustreError: 290447:0:(mgs_handler.c:517:mgs_target_reg()) fs22207-MDT0001: Error checking target index: rc = -13 [10272.087593] LustreError: fs22207-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10272.090898] LustreError: 294194:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10272.093636] LustreError: 294194:0:(tgt_mount.c:2128:server_put_super()) no obd fs22207-MDT0001 [10272.096370] LustreError: 294194:0:(tgt_mount.c:116:server_deregister_mount()) fs22207-MDT0001 not registered [10272.110248] Lustre: server umount fs22207-MDT0001 complete [10272.111524] LustreError: 294194:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10275.218740] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10275.248164] Lustre: Setting parameter fs22207-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs22207-MDT0001 [10275.252794] Lustre: Skipped 2 previous similar messages [10275.269202] Lustre: srv-fs22207-MDT0001: No data found on store. Initialize space. [10275.271474] Lustre: Skipped 4 previous similar messages [10275.313727] Lustre: fs22207-MDT0001: new disk, initializing [10275.315783] Lustre: Skipped 3 previous similar messages [10275.351015] Lustre: srv-fs22207-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10275.354056] Lustre: 294659:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22207-MDT0001-mdtlov: can't create new llog: rc = -115 [10275.357554] LustreError: 294659:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22207-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10276.777976] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10279.891160] Lustre: server umount fs22207-MDT0001 complete [10284.515847] loop1: detected capacity change from 0 to 204800000 [10286.704170] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10286.836511] Lustre: 296227:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22207-MDT0001-mdtlov: can't create new llog: rc = -115 [10286.839829] LustreError: 296227:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22207-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10288.390117] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10290.145624] Lustre: server umount fs22207-MDT0001 complete [10294.753088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10294.757268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10294.763329] Lustre: Skipped 2 previous similar messages [10297.313772] LustreError: 292284:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10297.320517] LustreError: 292284:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 25 previous similar messages [10299.870449] Lustre: server umount lustre-MDT0000 complete [10303.153879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10303.206751] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10304.845950] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10307.773275] loop1: detected capacity change from 0 to 204800000 [10308.585477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10309.941906] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10310.084524] Lustre: 298671:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22207-MDT0001-mdtlov: can't create new llog: rc = -115 [10310.089090] LustreError: 298671:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22207-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10311.625617] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10313.603823] Lustre: server umount fs22207-MDT0001 complete [10323.944880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10327.711226] Lustre: server umount lustre-OST0000 complete [10334.181398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10334.183994] Lustre: Skipped 11 previous similar messages [10335.414867] Lustre: server umount lustre-MDT0000 complete [10337.128829] LustreError: 290441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307057 with bad export cookie 2383986779212838694 [10337.132276] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10337.135336] LustreError: 290441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10337.299201] Lustre: server umount lustre-MDT0001 complete [10342.122949] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [10343.479774] Key type lgssc unregistered [10343.639511] LNet: 301009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10343.643936] LNetError: 301009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10344.678635] LNet: Removed LNI 192.168.204.157@tcp [10345.077448] Key type .llcrypt unregistered [10345.083364] Key type ._llcrypt unregistered [10354.338763] Key type ._llcrypt registered [10354.343682] Key type .llcrypt registered [10354.397159] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [10354.770408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10354.778845] alg: No test for adler32 (adler32-zlib) [10355.689025] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [10355.795378] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [10357.384738] Key type lgssc registered [10357.852678] Lustre: Echo OBD driver; http://www.lustre.org/ [10361.031491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10361.040870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10362.238856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10363.920797] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10367.322245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10367.510616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10369.127400] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10372.045916] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10374.271659] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10376.818664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10376.976529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10378.024876] LustreError: 304233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10378.033143] LustreError: 304233:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10379.334345] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10382.176642] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10383.332342] LustreError: 304080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10388.453435] LustreError: 304233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10388.463285] LustreError: 304233:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10392.602439] LustreError: 304233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10397.712266] LustreError: 304080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10397.719191] LustreError: 304080:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10399.584244] loop1: detected capacity change from 0 to 204800000 [10399.646854] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10399.651570] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10399.657120] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10401.580376] loop1: detected capacity change from 0 to 204800000 [10403.628039] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10403.694778] Lustre: fs1195-OST0000: new disk, initializing [10403.696787] Lustre: srv-fs1195-OST0000: No data found on store. Initialize space. [10403.699464] Lustre: 306003:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10403.730155] Lustre: fs1195-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10405.984260] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10407.951176] LustreError: 304078:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10407.958479] LustreError: 304078:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10409.742813] Lustre: server umount fs1195-OST0000 complete [10414.461602] loop1: detected capacity change from 0 to 204800000 [10416.774667] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10416.869694] Lustre: fs1195-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10419.221272] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10422.775418] loop2: detected capacity change from 0 to 204800000 [10422.861452] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10422.865650] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10422.870838] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10424.297770] LustreError: 304080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10424.305810] LustreError: 304080:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [10424.912981] loop2: detected capacity change from 0 to 204800000 [10426.966230] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10426.999431] LustreError: fs1195-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10427.005428] LustreError: fs1195-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10427.009806] LustreError: 309040:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10427.012708] LustreError: 309040:0:(tgt_mount.c:2128:server_put_super()) no obd fs1195-OST0001 [10427.015362] LustreError: 309040:0:(tgt_mount.c:116:server_deregister_mount()) fs1195-OST0001 not registered [10427.027559] Lustre: server umount fs1195-OST0001 complete [10427.029707] LustreError: 309040:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10430.796994] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10430.859976] Lustre: fs1195-OST0001: new disk, initializing [10430.863986] Lustre: srv-fs1195-OST0001: No data found on store. Initialize space. [10430.866753] Lustre: 309590:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10430.900842] Lustre: fs1195-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10433.298421] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10437.157808] Lustre: server umount fs1195-OST0000 complete [10440.714092] Lustre: server umount fs1195-OST0001 complete [10445.597909] loop1: detected capacity change from 0 to 204800000 [10447.746868] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10447.821291] Lustre: fs1195-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10450.073617] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10452.640326] loop2: detected capacity change from 0 to 204800000 [10454.823210] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10457.136817] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10459.149726] LustreError: 304078:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10459.157278] LustreError: 304078:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 18 previous similar messages [10460.300984] Lustre: server umount fs1195-OST0000 complete [10463.688042] Lustre: server umount fs1195-OST0001 complete [10470.369948] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10470.372769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10470.373691] LustreError: Skipped 1 previous similar message [10470.380046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10470.384348] Lustre: Skipped 1 previous similar message [10475.490802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10475.492884] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10475.493352] Lustre: Skipped 1 previous similar message [10480.609569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10480.611956] Lustre: Skipped 1 previous similar message [10482.144156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10482.212288] Lustre: server umount lustre-OST0000 complete [10485.217333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10485.221659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10485.227390] Lustre: Skipped 1 previous similar message [10485.229364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10490.849434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10490.851539] Lustre: Skipped 2 previous similar messages [10498.016129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10498.087401] Lustre: server umount lustre-MDT0000 complete [10499.388624] LustreError: 302235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307219 with bad export cookie 4322484988364891384 [10499.391469] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10499.393064] LustreError: 302235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10499.513216] Lustre: server umount lustre-MDT0001 complete [10503.283717] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [10504.382882] Key type lgssc unregistered [10504.513328] LNet: 315507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10504.515774] LNetError: 315507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10504.522288] LNet: Removed LNI 192.168.204.157@tcp [10504.873188] Key type .llcrypt unregistered [10504.874795] Key type ._llcrypt unregistered [10515.760090] Key type ._llcrypt registered [10515.761342] Key type .llcrypt registered [10515.802180] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [10522.636795] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [10522.987940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10523.011108] alg: No test for adler32 (adler32-zlib) [10523.882592] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [10523.987824] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [10525.584673] Key type lgssc registered [10526.042966] Lustre: Echo OBD driver; http://www.lustre.org/ [10540.084702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10540.091360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10541.180490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10541.193746] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10541.230820] Lustre: lustre-MDT0000: new disk, initializing [10541.256320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10541.264044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10542.773625] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10547.239538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10547.300251] Lustre: 319333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10547.328456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10547.330779] Lustre: Skipped 1 previous similar message [10547.369633] Lustre: lustre-MDT0001: new disk, initializing [10547.410946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10547.426776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10547.431087] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10548.962183] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10552.886871] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10554.865152] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10557.359218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10557.469976] Lustre: lustre-OST0000: new disk, initializing [10557.472209] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10557.475821] Lustre: 321300:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10557.507192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10559.769278] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10563.835564] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10566.777795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10566.781947] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10566.820187] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10572.275185] Lustre: Mounted lustre-client - version 2.17.58_39_gbf2c64c [10572.649881] Lustre: Unmounted lustre-client [10577.378025] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10577.378600] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10577.381015] LustreError: Skipped 1 previous similar message [10577.388339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10577.392189] Lustre: Skipped 1 previous similar message [10578.077840] Lustre: server umount lustre-OST0000 complete [10582.497828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10582.498573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10582.503029] Lustre: Skipped 2 previous similar messages [10585.588893] Lustre: server umount lustre-MDT0000 complete [10587.204658] LustreError: 319325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307307 with bad export cookie 11939523571201490743 [10587.207963] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10587.213711] LustreError: 319325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10587.347472] Lustre: server umount lustre-MDT0001 complete [10591.597921] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [10592.721853] Key type lgssc unregistered [10615.392326] LNet: 323457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10615.395539] LNetError: 323457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10696.294420] LNet: Removed LNI 192.168.204.157@tcp [10696.626336] Key type .llcrypt unregistered [10696.628104] Key type ._llcrypt unregistered [10703.131307] Key type ._llcrypt registered [10703.132739] Key type .llcrypt registered [10703.173758] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [10710.221766] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [10710.682786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10710.693928] alg: No test for adler32 (adler32-zlib) [10711.564976] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [10711.662802] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [10713.256215] Key type lgssc registered [10713.610470] Lustre: Echo OBD driver; http://www.lustre.org/ [10724.669600] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 09:50:43 (1789307443) [10727.081959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10727.088475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10728.179151] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10728.190650] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10728.226138] Lustre: lustre-MDT0000: new disk, initializing [10728.252534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10728.260083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10729.783518] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10734.196836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10734.243312] Lustre: 326650:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10734.266325] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10734.267946] Lustre: Skipped 1 previous similar message [10734.305408] Lustre: lustre-MDT0001: new disk, initializing [10734.339478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10734.351964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10734.355971] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10735.816469] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10739.609295] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10741.432835] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10743.677591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10743.769023] Lustre: lustre-OST0000: new disk, initializing [10743.770920] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10743.773944] Lustre: 328620:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10743.802471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10744.066402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10744.070064] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10744.111864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10745.806765] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10749.498794] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10757.137279] Lustre: server umount lustre-OST0000 complete [10759.649693] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10759.650421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10759.655039] Lustre: Skipped 1 previous similar message [10764.768951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10764.771093] Lustre: Skipped 2 previous similar messages [10768.894787] Lustre: server umount lustre-MDT0000 complete [10770.138428] LustreError: 327446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307490 with bad export cookie 10518620224998973825 [10770.142263] LustreError: 327446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10770.142312] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10780.693727] Lustre: server umount lustre-MDT0001 complete [10784.275760] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [10785.367262] Key type lgssc unregistered [10785.509464] LNet: 330524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10785.512294] LNetError: 330524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10785.522361] LNet: Removed LNI 192.168.204.157@tcp [10785.870464] Key type .llcrypt unregistered [10785.871949] Key type ._llcrypt unregistered [10793.826839] Key type ._llcrypt registered [10793.828412] Key type .llcrypt registered [10793.862560] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [10794.230974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10794.244770] alg: No test for adler32 (adler32-zlib) [10795.099831] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [10795.183258] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [10796.768183] Key type lgssc registered [10797.083752] Lustre: Echo OBD driver; http://www.lustre.org/ [10799.496441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10799.502635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10800.634991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10801.959612] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10804.827452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10804.952224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10806.347807] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10808.829990] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10810.730139] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10813.032699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10813.149964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10815.170348] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10817.651959] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10831.705455] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10832.265890] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10835.938197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10835.938740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10835.939520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10835.943558] Lustre: Skipped 1 previous similar message [10838.548290] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.57@tcp (stopping) [10838.550111] Lustre: Skipped 2 previous similar messages [10839.828914] Lustre: server umount lustre-MDT0000 complete [10841.057894] LustreError: 331771:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10841.063332] LustreError: 331771:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10841.118286] LustreError: 331752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307561 with bad export cookie 3391851973565687840 [10841.123510] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10843.662242] LustreError: 333608:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10843.668277] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.57@tcp (stopping) [10846.176882] LustreError: 333201:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10846.177265] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10846.183382] Lustre: Skipped 1 previous similar message [10846.184959] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10848.781417] LustreError: 331766:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10851.298180] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10851.299964] Lustre: Skipped 1 previous similar message [10853.901722] LustreError: 331765:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10853.907257] LustreError: 331765:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10855.392204] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10855.466713] Lustre: server umount lustre-MDT0001 complete [10862.801198] Lustre: server umount lustre-OST0000 complete [10865.613418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10867.025457] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10867.961155] Lustre: 335967:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.157@tcp [10868.673768] Lustre: server umount MGS complete [10871.372286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10871.514191] LustreError: 336538:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10871.534204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10872.906687] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10875.709532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10875.816561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10877.147445] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10879.958727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10882.062594] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10884.461681] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10884.974165] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [10886.663201] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10887.211428] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10890.721761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10890.722047] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10890.726903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10890.730081] Lustre: Skipped 2 previous similar messages [10905.056104] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10905.113429] Lustre: server umount lustre-OST0000 complete [10906.592857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10906.595203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10906.599951] Lustre: Skipped 1 previous similar message [10911.201181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10911.201614] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10911.203701] Lustre: Skipped 5 previous similar messages [10912.536936] Lustre: server umount lustre-MDT0000 complete [10913.822992] LustreError: 336518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307633 with bad export cookie 3391851973565688666 [10913.826211] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10913.826927] LustreError: 336518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10913.931558] Lustre: server umount lustre-MDT0001 complete [10917.426162] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [10918.451846] Key type lgssc unregistered [10918.583491] LNet: 339972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10918.587370] LNetError: 339972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10918.599359] LNet: Removed LNI 192.168.204.157@tcp [10918.932175] Key type .llcrypt unregistered [10918.933855] Key type ._llcrypt unregistered [10925.169399] Key type ._llcrypt registered [10925.170669] Key type .llcrypt registered [10925.212492] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [10931.593318] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [10931.952424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10931.987242] alg: No test for adler32 (adler32-zlib) [10932.850275] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [10932.936174] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [10934.520162] Key type lgssc registered [10934.855676] Lustre: Echo OBD driver; http://www.lustre.org/ [10946.646236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10946.651433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10947.729399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10947.738874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10947.771408] Lustre: lustre-MDT0000: new disk, initializing [10947.791102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10947.796874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10949.239636] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10953.682282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10953.726209] Lustre: 343099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10953.753284] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10953.756299] Lustre: Skipped 1 previous similar message [10953.796237] Lustre: lustre-MDT0001: new disk, initializing [10953.845179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10953.864659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10953.870282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10955.351981] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10959.280697] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10961.265385] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10963.584628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10963.683158] Lustre: lustre-OST0000: new disk, initializing [10963.685175] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10963.688362] Lustre: 345068:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10963.717543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10965.652116] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [10969.451394] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10970.104823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10970.108319] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10970.140864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10970.802347] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10970.868808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10972.237285] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10972.305689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10975.201706] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10975.202948] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10975.203913] LustreError: Skipped 1 previous similar message [10975.207997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10975.210552] Lustre: Skipped 1 previous similar message [10979.040204] Lustre: server umount lustre-OST0000 complete [10980.323210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10980.329183] Lustre: Skipped 1 previous similar message [10980.331494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10980.333514] Lustre: Skipped 1 previous similar message [10985.441397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10985.443526] Lustre: Skipped 1 previous similar message [10986.458944] Lustre: server umount lustre-MDT0000 complete [10987.714099] LustreError: 343897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307707 with bad export cookie 18335650951963941398 [10987.717146] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10987.717743] LustreError: 343897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10987.837853] Lustre: server umount lustre-MDT0001 complete [10990.809875] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 09:55:10 (1789307710) [10996.287574] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [10999.563532] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [11011.290576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11011.396490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11011.409538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11011.447046] Lustre: lustre-MDT0000: new disk, initializing [11011.471601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11011.479545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11012.868882] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11017.170356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11017.210066] Lustre: 349821:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11017.238917] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11017.241548] Lustre: Skipped 1 previous similar message [11017.279791] Lustre: lustre-MDT0001: new disk, initializing [11017.316535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11017.329403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11017.332494] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11018.753323] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11022.666655] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11024.684467] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11027.101213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11027.226724] Lustre: lustre-OST0000: new disk, initializing [11027.229471] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11027.233419] Lustre: 351787:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11027.269654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11028.024907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11028.029448] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11028.057156] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11029.428725] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11033.566653] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11048.418781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11048.420138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11048.420869] LustreError: Skipped 1 previous similar message [11048.425842] Lustre: Skipped 1 previous similar message [11048.428235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11048.431961] Lustre: Skipped 1 previous similar message [11049.615337] Lustre: server umount lustre-OST0000 complete [11053.025751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11053.029362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11053.033827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11057.051497] Lustre: server umount lustre-MDT0000 complete [11058.287585] LustreError: 350614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307778 with bad export cookie 18335650951963942266 [11058.291088] LustreError: 350614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11058.291188] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11058.412457] Lustre: server umount lustre-MDT0001 complete [11062.005059] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [11063.005596] Key type lgssc unregistered [11063.151548] LNet: 353932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11063.154865] LNetError: 353932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11063.162480] LNet: Removed LNI 192.168.204.157@tcp [11063.475164] Key type .llcrypt unregistered [11063.476280] Key type ._llcrypt unregistered [11070.825185] Key type ._llcrypt registered [11070.826238] Key type .llcrypt registered [11070.863201] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [11071.265484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11071.317352] alg: No test for adler32 (adler32-zlib) [11072.169992] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [11072.270413] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [11073.848158] Key type lgssc registered [11074.201493] Lustre: Echo OBD driver; http://www.lustre.org/ [11076.770839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11076.776348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11077.913039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11079.322991] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11082.221881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11082.339138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11083.638053] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11085.980994] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11087.805552] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11090.129258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11090.242697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11092.243099] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11094.691169] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11123.172127] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11123.172387] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11123.179881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11123.181871] Lustre: Skipped 1 previous similar message [11128.292578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11128.294405] Lustre: Skipped 1 previous similar message [11128.659178] Lustre: server umount lustre-OST0000 complete [11133.409604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11133.409722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11133.412248] LustreError: Skipped 1 previous similar message [11133.416783] Lustre: Skipped 1 previous similar message [11133.420726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11133.423113] Lustre: Skipped 1 previous similar message [11138.529096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11138.531641] Lustre: Skipped 1 previous similar message [11143.649038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11143.651115] Lustre: Skipped 1 previous similar message [11144.160083] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11144.218168] Lustre: server umount lustre-MDT0000 complete [11145.480343] LustreError: 355161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789307865 with bad export cookie 11464427953084602391 [11145.483650] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11145.484626] LustreError: 355161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11145.592935] Lustre: server umount lustre-MDT0001 complete [11149.421302] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [11150.504631] Key type lgssc unregistered [11150.644359] LNet: 360798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11150.647509] LNetError: 360798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11150.658416] LNet: Removed LNI 192.168.204.157@tcp [11150.996642] Key type .llcrypt unregistered [11150.997856] Key type ._llcrypt unregistered [11183.996070] Key type ._llcrypt registered [11183.997293] Key type .llcrypt registered [11184.032742] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [11184.125473] Key type .llcrypt unregistered [11184.127075] Key type ._llcrypt unregistered [11210.794629] Key type ._llcrypt registered [11210.795638] Key type .llcrypt registered [11210.831099] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [11210.927253] Key type .llcrypt unregistered [11210.928943] Key type ._llcrypt unregistered [11221.723967] Key type ._llcrypt registered [11221.725014] Key type .llcrypt registered [11221.760162] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [11221.855336] Key type .llcrypt unregistered [11221.856784] Key type ._llcrypt unregistered [11231.982123] Key type ._llcrypt registered [11231.983402] Key type .llcrypt registered [11232.015051] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [11232.102234] Key type .llcrypt unregistered [11232.103661] Key type ._llcrypt unregistered [11258.866779] Key type ._llcrypt registered [11258.867663] Key type .llcrypt registered [11258.900133] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [11259.008213] Key type .llcrypt unregistered [11259.009096] Key type ._llcrypt unregistered [11269.978909] Key type ._llcrypt registered [11269.979742] Key type .llcrypt registered [11270.013201] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [11270.114222] Key type .llcrypt unregistered [11270.115081] Key type ._llcrypt unregistered [11281.304309] Key type ._llcrypt registered [11281.305609] Key type .llcrypt registered [11281.343451] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [11281.440459] Key type .llcrypt unregistered [11281.442180] Key type ._llcrypt unregistered [11292.548386] Key type ._llcrypt registered [11292.549387] Key type .llcrypt registered [11292.583490] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [11292.690218] Key type .llcrypt unregistered [11292.691554] Key type ._llcrypt unregistered [11304.323393] Key type ._llcrypt registered [11304.324468] Key type .llcrypt registered [11304.363976] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [11304.465189] Key type .llcrypt unregistered [11304.466217] Key type ._llcrypt unregistered [11314.982908] Key type ._llcrypt registered [11314.983916] Key type .llcrypt registered [11315.022430] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [11315.117392] Key type .llcrypt unregistered [11315.118489] Key type ._llcrypt unregistered [11336.435954] Key type ._llcrypt registered [11336.437438] Key type .llcrypt registered [11336.476748] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [11336.581574] Key type .llcrypt unregistered [11336.582985] Key type ._llcrypt unregistered [11347.562731] Key type ._llcrypt registered [11347.563695] Key type .llcrypt registered [11347.600878] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [11347.700352] Key type .llcrypt unregistered [11347.701261] Key type ._llcrypt unregistered [11358.838898] Key type ._llcrypt registered [11358.839890] Key type .llcrypt registered [11358.876424] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [11358.978351] Key type .llcrypt unregistered [11358.979401] Key type ._llcrypt unregistered [11376.187217] Key type ._llcrypt registered [11376.188615] Key type .llcrypt registered [11376.222626] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [11382.338381] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [11382.635408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11382.643221] alg: No test for adler32 (adler32-zlib) [11383.491352] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [11383.570099] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [11385.152193] Key type lgssc registered [11385.529441] Lustre: Echo OBD driver; http://www.lustre.org/ [11397.014238] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11397.022827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11398.112557] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [11398.124650] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [11398.162197] Lustre: tst250fs-MDT0000: new disk, initializing [11398.192703] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11398.200864] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11399.641247] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11403.823400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11403.869856] Lustre: 380655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11403.892777] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [11403.895058] Lustre: Skipped 1 previous similar message [11403.927625] Lustre: tst250fs-MDT0001: new disk, initializing [11403.956146] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11403.967458] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11403.971762] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11405.389670] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11407.770164] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11410.199576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11410.314056] Lustre: tst250fs-OST0000: new disk, initializing [11410.316302] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [11410.320141] Lustre: 382285:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11410.354814] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11412.432718] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11412.545056] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11412.548814] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11412.591777] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11416.956328] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11417.030684] Lustre: tst250fs-OST0001: new disk, initializing [11417.032924] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [11417.036678] Lustre: 383160:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11417.073817] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11418.760720] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11418.764811] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11418.800479] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11419.200860] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [11421.812221] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [11434.467055] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11434.468764] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11434.471552] Lustre: Skipped 2 previous similar messages [11434.474962] Lustre: Skipped 3 previous similar messages [11439.586251] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11439.587898] Lustre: Skipped 2 previous similar messages [11444.706122] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11444.709222] Lustre: Skipped 3 previous similar messages [11446.240110] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11446.298197] Lustre: server umount tst250fs-MDT0000 complete [11447.499391] LustreError: 380646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789308167 with bad export cookie 7930204617492885094 [11447.503918] LustreError: 380646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11447.504815] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11447.623401] Lustre: server umount tst250fs-MDT0001 complete [11459.014559] Lustre: server umount tst250fs-OST0000 complete [11470.556946] Lustre: server umount tst250fs-OST0001 complete [11478.004120] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [11480.904484] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [11491.909831] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 10:03:31 (1789308211) [11492.492947] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [11497.979251] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [11501.657206] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [11514.221230] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11227 sec ========= 10:03:53 (1789308233) [11514.974216] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:03:54 (1789308234) === [11516.394842] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:03:55 (1789308235) === [11531.608600] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [11532.618551] Key type lgssc unregistered [11532.735345] LNet: 392268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11532.737506] LNetError: 392268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11532.745280] LNet: Removed LNI 192.168.204.157@tcp [11533.038123] Key type .llcrypt unregistered [11533.039325] Key type ._llcrypt unregistered