[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 493801475 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003121] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.007756] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.010129] LSM: Security Framework initializing [ 0.011065] Yama: becoming mindful. [ 0.012039] SELinux: Initializing. [ 0.013084] *** VALIDATE selinux *** [ 0.021754] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025885] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027069] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028138] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029132] *** VALIDATE tmpfs *** [ 0.031411] *** VALIDATE proc *** [ 0.033224] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035285] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037133] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039033] Spectre V2 : User space: Vulnerable [ 0.040012] Speculative Store Bypass: Vulnerable [ 0.042948] debug: unmapping init [mem 0xffffffff95a59000-0xffffffff95a60fff] [ 0.044150] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045690] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047022] ... bit width: 48 [ 0.048018] ... generic registers: 4 [ 0.049014] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051019] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053303] rcu: Hierarchical SRCU implementation. [ 0.055419] smp: Bringing up secondary CPUs ... [ 0.056621] x86: Booting SMP configuration: [ 0.057033] .... node #0, CPUs: #1 #2 #3 [ 0.060663] smp: Brought up 1 node, 4 CPUs [ 0.062015] smpboot: Max logical packages: 1 [ 0.063020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138633] node 0 deferred pages initialised in 73ms [ 0.141121] devtmpfs: initialized [ 0.142224] x86/mm: Memory block size: 128MB [ 0.144850] gcov: version magic: 0x41383552 [ 0.146312] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.147095] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.148246] pinctrl core: initialized pinctrl subsystem [ 0.149223] [ 0.149847] ************************************************************* [ 0.150015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151012] ** ** [ 0.152012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.153013] ** ** [ 0.154015] ** This means that this kernel is built to expose internal ** [ 0.155017] ** IOMMU data structures, which may compromise security on ** [ 0.156012] ** your system. ** [ 0.157014] ** ** [ 0.158013] ** If you see this message and you are not debugging the ** [ 0.159014] ** kernel, report this immediately to your vendor! ** [ 0.160015] ** ** [ 0.161015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162014] ************************************************************* [ 0.163847] NET: Registered protocol family 16 [ 0.164463] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.165066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.166071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.168045] cpuidle: using governor menu [ 0.171082] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.175495] PCI: Using configuration type 1 for base access [ 0.179153] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.191141] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.193060] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.197182] cryptd: max_cpu_qlen set to 1000 [ 0.199286] ACPI: Added _OSI(Module Device) [ 0.200016] ACPI: Added _OSI(Processor Device) [ 0.201018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.202017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.206240] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.209577] ACPI: Interpreter enabled [ 0.210071] ACPI: PM: (supports S0 S3 S4 S5) [ 0.211009] ACPI: Using IOAPIC for interrupt routing [ 0.212120] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.213373] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.222807] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.223034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.224019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.225063] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.227216] acpiphp: Slot [2] registered [ 0.228114] acpiphp: Slot [5] registered [ 0.229105] acpiphp: Slot [6] registered [ 0.230135] acpiphp: Slot [7] registered [ 0.231128] acpiphp: Slot [8] registered [ 0.232102] acpiphp: Slot [9] registered [ 0.233093] acpiphp: Slot [10] registered [ 0.235119] acpiphp: Slot [3] registered [ 0.236072] acpiphp: Slot [4] registered [ 0.238141] acpiphp: Slot [11] registered [ 0.240130] acpiphp: Slot [12] registered [ 0.241103] acpiphp: Slot [13] registered [ 0.243120] acpiphp: Slot [14] registered [ 0.244098] acpiphp: Slot [15] registered [ 0.246098] acpiphp: Slot [16] registered [ 0.247102] acpiphp: Slot [17] registered [ 0.248094] acpiphp: Slot [18] registered [ 0.250106] acpiphp: Slot [19] registered [ 0.251113] acpiphp: Slot [20] registered [ 0.253105] acpiphp: Slot [21] registered [ 0.254105] acpiphp: Slot [22] registered [ 0.255095] acpiphp: Slot [23] registered [ 0.257095] acpiphp: Slot [24] registered [ 0.258100] acpiphp: Slot [25] registered [ 0.261103] acpiphp: Slot [26] registered [ 0.262106] acpiphp: Slot [27] registered [ 0.264108] acpiphp: Slot [28] registered [ 0.265088] acpiphp: Slot [29] registered [ 0.267101] acpiphp: Slot [30] registered [ 0.268081] acpiphp: Slot [31] registered [ 0.269102] PCI host bridge to bus 0000:00 [ 0.271025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.273022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.276029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.278024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.281023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.284026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.286189] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.288993] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.292270] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.303867] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.310128] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.313019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.315019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.318020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.322468] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.325723] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.329045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.332682] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.338013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.351015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.357013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.364406] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.369014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.375014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.390017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.399818] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.408016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.421014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.445021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.459124] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.465013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.473014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.495017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.504234] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.511015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.519017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.535016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.545475] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.551016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.560015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.579017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.591035] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.599014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.606021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.628019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.643878] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.646383] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.649411] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.652439] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.655220] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.660190] iommu: Default domain type: Passthrough [ 0.662491] SCSI subsystem initialized [ 0.664166] ACPI: bus type USB registered [ 0.666100] usbcore: registered new interface driver usbfs [ 0.668085] usbcore: registered new interface driver hub [ 0.670088] usbcore: registered new device driver usb [ 0.671207] pps_core: LinuxPPS API ver. 1 registered [ 0.674013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.676055] PTP clock support registered [ 0.678153] EDAC MC: Ver: 3.0.0 [ 0.680146] PCI: Using ACPI for IRQ routing [ 0.682985] NetLabel: Initializing [ 0.684010] NetLabel: domain hash size = 128 [ 0.685009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.687082] NetLabel: unlabeled traffic allowed by default [ 0.689118] vgaarb: loaded [ 0.690249] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.692010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.698575] clocksource: Switched to clocksource kvm-clock [ 0.813444] VFS: Disk quotas dquot_6.6.0 [ 0.815758] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.819587] *** VALIDATE ramfs *** [ 0.821503] *** VALIDATE hugetlbfs *** [ 0.823900] pnp: PnP ACPI init [ 0.826981] pnp: PnP ACPI: found 6 devices [ 0.852549] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.854783] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.856617] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.858882] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.861034] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.863494] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.866308] NET: Registered protocol family 2 [ 0.869183] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.874485] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.880123] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.884692] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.887274] TCP: Hash tables configured (established 65536 bind 65536) [ 0.889369] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.891501] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.893892] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.896729] NET: Registered protocol family 1 [ 0.899975] RPC: Registered named UNIX socket transport module. [ 0.902048] RPC: Registered udp transport module. [ 0.903253] RPC: Registered tcp transport module. [ 0.904506] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.906088] NET: Registered protocol family 44 [ 0.907736] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.909800] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.911946] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.914105] PCI: CLS 0 bytes, default 64 [ 0.915956] Unpacking initramfs... [ 2.325538] debug: unmapping init [mem 0xffff8a56bcc54000-0xffff8a56bffbffff] [ 2.330359] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.332779] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.335356] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.829576] Initialise system trusted keyrings [ 2.831585] Key type blacklist registered [ 2.833610] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.843525] zbud: loaded [ 2.846723] *** VALIDATE nfs *** [ 2.848085] *** VALIDATE nfs4 *** [ 2.850021] pstore: using deflate compression [ 2.855802] Platform Keyring initialized [ 2.965314] NET: Registered protocol family 38 [ 2.967161] Key type asymmetric registered [ 2.969088] Asymmetric key parser 'x509' registered [ 2.971233] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.975143] io scheduler mq-deadline registered [ 2.977035] io scheduler kyber registered [ 2.978792] io scheduler bfq registered [ 2.980955] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.984490] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.987719] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.991139] ACPI: Power Button [PWRF] [ 2.997032] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.003995] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.022534] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.032770] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.047462] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.076407] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.103472] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.108632] Non-volatile memory driver v1.3 [ 3.110638] Linux agpgart interface v0.103 [ 3.142681] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.145629] vda: detected capacity change from 0 to 74698752 [ 3.164619] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.167351] vdb: detected capacity change from 0 to 1073741824 [ 3.187131] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.190536] vdc: detected capacity change from 0 to 2621440000 [ 3.208198] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.211896] vdd: detected capacity change from 0 to 2621440000 [ 3.229483] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.232585] vde: detected capacity change from 0 to 4294967296 [ 3.249941] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.253129] vdf: detected capacity change from 0 to 4294967296 [ 3.264267] libphy: Fixed MDIO Bus: probed [ 3.274161] usbcore: registered new interface driver usbserial_generic [ 3.276335] usbserial: USB Serial support registered for generic [ 3.278736] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.283699] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.285411] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.287922] mousedev: PS/2 mouse device common for all mice [ 3.290902] rtc_cmos 00:05: RTC can wake from S4 [ 3.294112] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.295982] rtc_cmos 00:05: registered as rtc0 [ 3.298953] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.302361] intel_pstate: CPU model not supported [ 3.305949] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.306320] hid: raw HID events driver (C) Jiri Kosina [ 3.311483] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.311802] usbcore: registered new interface driver usbhid [ 3.316075] usbhid: USB HID core driver [ 3.317327] drop_monitor: Initializing network drop monitor service [ 3.319197] Initializing XFRM netlink socket [ 3.320919] NET: Registered protocol family 10 [ 3.323776] Segment Routing with IPv6 [ 3.325462] NET: Registered protocol family 17 [ 3.327102] mpls_gso: MPLS GSO support [ 3.334338] RAS: Correctable Errors collector initialized. [ 3.336728] AVX version of gcm_enc/dec engaged. [ 3.338443] AES CTR mode by8 optimization enabled [ 3.411573] sched_clock: Marking stable (3411549781, 0)->(4403685350, -992135569) [ 3.415546] registered taskstats version 1 [ 3.417748] Loading compiled-in X.509 certificates [ 3.419681] zswap: loaded using pool lzo/zbud [ 3.441883] Key type big_key registered [ 3.454161] Key type encrypted registered [ 3.455708] ima: No TPM chip found, activating TPM-bypass! [ 3.458765] ima: Allocated hash algorithm: sha1 [ 3.460117] ima: No architecture policies found [ 3.461622] evm: Initialising EVM extended attributes: [ 3.463107] evm: security.selinux [ 3.464140] evm: security.ima [ 3.465093] evm: security.capability [ 3.466182] evm: HMAC attrs: 0x1 [ 3.468407] rtc_cmos 00:05: setting system clock to 2026-09-03 14:01:46 UTC (1788444106) [ 3.474355] debug: unmapping init [mem 0xffffffff96a03000-0xffffffff96bfffff] [ 3.477231] debug: unmapping init [mem 0xffffffff95782000-0xffffffff95a58fff] [ 3.485069] Write protecting the kernel read-only data: 28672k [ 3.489069] debug: unmapping init [mem 0xffffffff93e03000-0xffffffff93ffffff] [ 3.491441] debug: unmapping init [mem 0xffffffff94714000-0xffffffff947fffff] [ 3.523072] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.532531] systemd[1]: Detected virtualization kvm. [ 3.534487] systemd[1]: Detected architecture x86-64. [ 3.536322] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.561348] systemd[1]: No hostname configured. [ 3.562619] systemd[1]: Set hostname to . [ 3.563916] random: systemd: uninitialized urandom read (16 bytes read) [ 3.566827] systemd[1]: Initializing machine ID from random generator. [ 3.698893] random: systemd: uninitialized urandom read (16 bytes read) [ 3.701473] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.705773] random: systemd: uninitialized urandom read (16 bytes read) [ 3.708726] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.712644] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.287238] device-mapper: uevent: version 1.0.3 [ 4.289615] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [[ 4.900098] random: fast init done  OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook...[ 4.946081] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target System Initialization. [ OK [ 5.002488] scsi host0: ata_piix ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.047207] scsi host1: ata_piix [ 5.048514] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.050368] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.606982] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.970616] random: crng init done [ 9.971961] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.306710] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.444448] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.696043] SELinux: Disabled at runtime. [ 11.755614] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.763758] systemd[1]: Detected virtualization kvm. [ 11.765770] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.215105] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.218289] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.223329] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.227336] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.230834] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.238813] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.243584] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ 12.337862] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.695090] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.986483] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.030116] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.130734] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.139699] EDAC sbridge: Ver: 1.1.2 [ 14.863270] Key type dns_resolver registered [ 15.174067] NFS: Registering the id_resolver key type [ 15.175996] Key type id_resolver registered [ 15.177401] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg450-server login: [ 41.038950] libcfs: loading out-of-tree module taints kernel. [ 41.060203] Key type ._llcrypt registered [ 41.062648] Key type .llcrypt registered [ 41.113843] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 56.191306] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 57.818760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 57.834256] alg: No test for adler32 (adler32-zlib) [ 59.291815] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 60.221535] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 62.023318] Key type lgssc registered [ 63.747369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 77.008617] hrtimer: interrupt took 7107942 ns [ 80.002957] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 123.949176] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 137.390790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 137.420936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 138.847333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 138.879808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 139.001261] Lustre: lustre-MDT0000: new disk, initializing [ 139.090563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 139.114260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 143.878427] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 159.776464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 159.917941] Lustre: 6506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 159.955711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 159.966884] Lustre: Skipped 1 previous similar message [ 160.055626] Lustre: lustre-MDT0001: new disk, initializing [ 160.112074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 160.153037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 160.166225] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 165.501155] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.788926] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 181.689079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 182.001248] Lustre: lustre-OST0000: new disk, initializing [ 182.008305] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 182.021945] Lustre: 8445:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 182.103661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 188.552846] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.977278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 188.993873] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 189.067051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 203.915576] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 204.121567] Lustre: lustre-OST0001: new disk, initializing [ 204.134576] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 204.162501] Lustre: 9520:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 204.276550] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 211.280271] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.122925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 212.159714] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 212.285364] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 224.206569] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 231.481603] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 238.627846] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing check_logdir /tmp/testlogs/ [ 243.778533] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing yml_node [ 248.442089] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 251.115688] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 253.865968] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 255.459253] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 10:05:57 EDT 2026 [ 276.426907] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 279.076544] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 298.465434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 298.473594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.505293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 299.490854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 299.491281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 299.508095] Lustre: Skipped 2 previous similar messages [ 299.527033] Lustre: Skipped 2 previous similar messages [ 302.039453] Lustre: server umount lustre-MDT0000 complete [ 306.350958] LustreError: 7455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444409 with bad export cookie 2860838255185656307 [ 306.369271] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 306.378979] LustreError: 7455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 309.727828] LustreError: 6513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 309.731532] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.740694] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 309.740715] Lustre: Skipped 1 previous similar message [ 309.754677] LustreError: 6513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 309.793534] Lustre: Skipped 1 previous similar message [ 311.778507] LustreError: 6513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 311.778543] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 311.814855] Lustre: Skipped 2 previous similar messages [ 313.002598] Lustre: server umount lustre-MDT0001 complete [ 317.224443] Lustre: server umount lustre-OST0000 complete [ 322.152137] Lustre: server umount lustre-OST0001 complete [ 328.169772] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 339.862986] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 376.164954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 376.445211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 376.541262] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 376.635258] Lustre: lustre-MDT0000: new disk, initializing [ 376.713273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 376.730911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 381.555200] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 394.322720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.484595] Lustre: 15073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 394.632133] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 394.655639] Lustre: Skipped 1 previous similar message [ 394.803433] Lustre: lustre-MDT0001: new disk, initializing [ 394.890353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 394.944534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 394.960332] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 399.820338] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 408.832874] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 415.290356] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 422.669154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 422.948939] Lustre: lustre-OST0000: new disk, initializing [ 422.956435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 422.968424] Lustre: 17041:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 423.061510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 424.301319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 424.317109] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 424.506417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 430.325870] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 440.209100] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 446.053484] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 446.288665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 451.955248] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 452.220818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 455.141191] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 455.146957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.149658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 455.168480] Lustre: Skipped 1 previous similar message [ 460.297489] Lustre: server umount lustre-OST0000 complete [ 465.381476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.383763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 465.396694] Lustre: Skipped 1 previous similar message [ 465.406148] Lustre: Skipped 2 previous similar messages [ 471.482763] Lustre: server umount lustre-MDT0000 complete [ 475.418372] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444578 with bad export cookie 2860838255185657840 [ 475.427897] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 475.430528] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 475.850829] Lustre: server umount lustre-MDT0001 complete [ 482.810644] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 10:09:44 (1788444584) [ 503.503429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 504.030825] LustreError: 19916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.057822] LustreError: 19916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 504.180797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 509.333444] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 509.419330] LustreError: 19917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.549740] LustreError: 19916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.244224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 523.985960] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 532.639575] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 538.804266] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 541.347031] Lustre: Failing over lustre-MDT0000 [ 541.727338] Lustre: server umount lustre-MDT0000 complete [ 543.393380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 543.414615] LustreError: 19911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 543.440563] LustreError: 19911:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 559.071126] Lustre: 3642:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444646/real 1788444646] req@ffff8a5603231180 x1875319626826240/t0(0) o400->MGC192.168.204.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788444662 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 559.094784] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 560.915122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.232926] LustreError: 19912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.280693] LustreError: 19912:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 570.596930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 570.604443] Lustre: Skipped 1 previous similar message [ 575.292954] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 575.975460] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 575.986458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 576.047803] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 587.768397] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 591.330499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 591.341936] LustreError: Skipped 1 previous similar message [ 591.345953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 591.360953] Lustre: Skipped 1 previous similar message [ 591.364805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 591.368933] Lustre: Skipped 2 previous similar messages [ 598.497410] LustreError: 19917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 598.513300] LustreError: 19917:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 599.676292] Lustre: server umount lustre-MDT0000 complete [ 604.175516] LustreError: 19898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444707 with bad export cookie 2860838255185659254 [ 604.187597] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 604.197164] LustreError: 19898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 604.513170] Lustre: server umount lustre-MDT0001 complete [ 614.369531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 620.157100] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 628.607987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 633.606614] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 640.910717] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 647.221174] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 649.774138] Lustre: Failing over lustre-MDT0000 [ 649.789096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 649.819167] Lustre: Skipped 1 previous similar message [ 649.832213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 649.845147] Lustre: Skipped 2 previous similar messages [ 650.259765] Lustre: server umount lustre-MDT0000 complete [ 654.820962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 666.080499] LustreError: 23258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 666.106207] LustreError: 23258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 669.910659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 670.079289] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 670.482826] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 675.448395] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 675.496725] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 675.512051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 675.521492] Lustre: Skipped 1 previous similar message [ 675.539990] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 687.211575] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 691.168084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 697.365599] Lustre: server umount lustre-MDT0000 complete [ 701.249359] LustreError: 23237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444804 with bad export cookie 2860838255185660192 [ 701.254249] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 701.260380] LustreError: 23237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 701.632763] Lustre: server umount lustre-MDT0001 complete [ 710.156860] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 10:13:31 (1788444811) [ 729.418164] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 741.622448] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 785.608885] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 797.455991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 797.876689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 797.944948] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 798.146797] Lustre: lustre-MDT0000: new disk, initializing [ 798.310658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 798.324955] Lustre: Skipped 3 previous similar messages [ 798.366642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 804.795969] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 817.564252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 817.764677] Lustre: 29632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 818.029639] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 823.206419] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 835.805368] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 842.916675] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 851.853624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 852.227805] Lustre: 31605:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 854.153395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 854.220690] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 854.309965] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 854.337461] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 860.397362] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 873.028717] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 880.471294] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 905.705052] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 905.709979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 905.714265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 905.714273] Lustre: Skipped 4 previous similar messages [ 905.719250] LustreError: Skipped 1 previous similar message [ 905.779604] Lustre: Skipped 4 previous similar messages [ 909.239850] Lustre: server umount lustre-OST0000 complete [ 928.223519] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 928.676651] Lustre: server umount lustre-MDT0000 complete [ 930.288593] LustreError: 29644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 930.303060] LustreError: 29644:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 934.330287] LustreError: 29624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445037 with bad export cookie 2860838255185660598 [ 934.338507] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 934.341748] LustreError: 29624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 934.959525] Lustre: server umount lustre-MDT0001 complete [ 947.253923] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 950.556585] Key type lgssc unregistered [ 950.898821] LNet: 33590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.923273] LNetError: 33590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.950786] LNet: Removed LNI 192.168.204.150@tcp [ 952.257449] Key type .llcrypt unregistered [ 952.259271] Key type ._llcrypt unregistered [ 980.855116] Key type ._llcrypt registered [ 980.858414] Key type .llcrypt registered [ 981.045468] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 982.438820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 982.525031] alg: No test for adler32 (adler32-zlib) [ 983.685677] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 984.087628] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 985.791153] Key type lgssc registered [ 987.146753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 995.981887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 996.020070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 997.526464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1002.445690] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1007.083043] LustreError: 35005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1007.106123] LustreError: 35005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1011.686136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1012.173288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1017.996864] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1026.703727] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1033.219505] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1040.983708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1041.465608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1048.454251] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1057.846456] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1058.315643] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1072.404394] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 10:19:34 (1788445174) [ 1074.519650] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1076.389901] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1084.269898] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 10:19:45 (1788445185) [ 1093.913757] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 10:19:55 (1788445195) [ 1095.160950] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1104.014962] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1104.021385] Lustre: Skipped 2 previous similar messages [ 1105.807881] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 10:20:07 (1788445207) [ 1109.235687] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1109.240493] Lustre: Skipped 3 previous similar messages [ 1128.736231] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1128.744961] Lustre: Skipped 6 previous similar messages [ 1132.443456] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 10:20:33 (1788445233) [ 1135.459857] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1139.083776] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1148.130375] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 10:20:49 (1788445249) [ 1150.453915] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1150.471916] Lustre: Skipped 3 previous similar messages [ 1164.071257] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 10:21:05 (1788445265) [ 1180.643576] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1180.647523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1180.651844] LustreError: Skipped 1 previous similar message [ 1180.653216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1180.702416] Lustre: Skipped 1 previous similar message [ 1185.100907] Lustre: server umount lustre-OST0000 complete [ 1190.883946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1190.890655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1190.898821] Lustre: Skipped 1 previous similar message [ 1190.916695] Lustre: Skipped 2 previous similar messages [ 1194.494802] Lustre: server umount lustre-MDT0000 complete [ 1195.999916] LustreError: 35000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1196.029669] LustreError: 35000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1198.866918] LustreError: 38568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445301 with bad export cookie 16387400116215925531 [ 1198.869324] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1198.879173] LustreError: 38568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1199.164554] Lustre: server umount lustre-MDT0001 complete [ 1211.929900] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 1214.883215] Key type lgssc unregistered [ 1215.161716] LNet: 42861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1215.166691] LNetError: 42861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1215.179934] LNet: Removed LNI 192.168.204.150@tcp [ 1216.157495] Key type .llcrypt unregistered [ 1216.165260] Key type ._llcrypt unregistered [ 1235.620582] Key type ._llcrypt registered [ 1235.624482] Key type .llcrypt registered [ 1235.805747] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 1252.026372] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 1252.897569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1253.051367] alg: No test for adler32 (adler32-zlib) [ 1254.145359] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1254.420573] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 1256.226109] Key type lgssc registered [ 1257.629836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1299.605616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1299.665232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1301.114596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1301.188761] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1301.304555] Lustre: lustre-MDT0000: new disk, initializing [ 1301.400328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1301.427253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1307.293639] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1319.900537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1320.040886] Lustre: 46012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1320.117263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1320.129715] Lustre: Skipped 1 previous similar message [ 1320.239974] Lustre: lustre-MDT0001: new disk, initializing [ 1320.431032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1320.487698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1320.499538] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1324.588923] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1334.097508] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1341.167599] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1349.436504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1349.974277] Lustre: lustre-OST0000: new disk, initializing [ 1349.982486] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1349.997821] Lustre: 47985:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1350.142630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1356.902072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1356.922744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1357.109804] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1358.564976] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1370.859650] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1375.610661] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1375.800770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1380.481037] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1380.734400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1382.882259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1382.884058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1382.903102] LustreError: Skipped 1 previous similar message [ 1382.905328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1382.953930] Lustre: Skipped 1 previous similar message [ 1388.005864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1388.017829] Lustre: Skipped 1 previous similar message [ 1388.799922] Lustre: server umount lustre-OST0000 complete [ 1396.194916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1396.217458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1396.223216] Lustre: Skipped 2 previous similar messages [ 1401.217985] Lustre: server umount lustre-MDT0000 complete [ 1405.503097] LustreError: 46006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445508 with bad export cookie 7165273088344329963 [ 1405.504725] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1405.517283] LustreError: 46006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1406.217233] Lustre: server umount lustre-MDT0001 complete [ 1416.659935] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 10:25:18 (1788445518) [ 1424.305279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1424.778197] LustreError: 50446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1424.796930] LustreError: 50446:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1424.880874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1429.995494] LustreError: 50447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1430.403318] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1434.151446] LustreError: 50446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1439.201908] LustreError: 50447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1440.155242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1440.767659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1446.381327] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1455.544760] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1461.338171] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1468.849858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1469.300463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1476.331655] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1484.460418] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1492.957300] LustreError: 52975:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1493.913076] Lustre: Modifying parameter general.timeout=1 in log params [ 1495.675965] Lustre: Modifying parameter general.timeout=3 in log params [ 1495.681500] Lustre: Skipped 1 previous similar message [ 1498.454991] Lustre: Modifying parameter general.timeout=6 in log params [ 1498.458751] Lustre: Skipped 2 previous similar messages [ 1503.186364] Lustre: Modifying parameter general.timeout=11 in log params [ 1503.190782] Lustre: Skipped 4 previous similar messages [ 1511.315199] Lustre: Modifying parameter general.timeout=21 in log params [ 1511.318835] Lustre: Skipped 9 previous similar messages [ 1528.198467] Lustre: Modifying parameter general.timeout=36 in log params [ 1528.201120] Lustre: Skipped 14 previous similar messages [ 1560.472499] Lustre: Modifying parameter general.timeout=68 in log params [ 1560.479745] Lustre: Skipped 31 previous similar messages [ 1631.738330] Lustre: Disabling parameter general.timeout= in log params [ 1631.750051] Lustre: Skipped 61 previous similar messages [ 1635.437743] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 10:28:56 (1788445736) [ 1639.126912] Lustre: 59925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1641.394524] Lustre: 60023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1645.236452] Lustre: 60234:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1645.248638] Lustre: 60234:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1649.361821] Lustre: 60444:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1649.378971] Lustre: 60444:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1678.688772] Lustre: 61801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1678.700391] LustreError: 61801:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1682.899205] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 10:29:44 (1788445784) [ 1696.238163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1696.241369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1696.255676] Lustre: Skipped 3 previous similar messages [ 1696.278339] Lustre: Skipped 2 previous similar messages [ 1701.349176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1701.359130] Lustre: Skipped 2 previous similar messages [ 1702.087867] Lustre: server umount lustre-MDT0000 complete [ 1706.104217] LustreError: 50427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445809 with bad export cookie 7165273088344330845 [ 1706.124912] LustreError: 50427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1706.126791] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1706.470717] Lustre: server umount lustre-MDT0001 complete [ 1721.054220] Lustre: server umount lustre-OST0000 complete [ 1752.370431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1752.762350] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1753.055873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1757.912605] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1767.147075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1767.262551] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1767.307301] Lustre: 64333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1772.443614] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1779.809474] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1785.099632] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1792.053731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1792.337486] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1792.687397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1792.695948] Lustre: Skipped 1 previous similar message [ 1799.990982] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1806.970881] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1808.424122] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1808.432785] Lustre: Skipped 23 previous similar messages [ 1808.455133] Lustre: 66898:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1819.110665] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1819.111233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1819.134644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1819.146972] Lustre: Skipped 1 previous similar message [ 1821.086231] Lustre: server umount lustre-OST0000 complete [ 1824.253724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1824.261142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1824.279792] Lustre: Skipped 2 previous similar messages [ 1829.784386] Lustre: server umount lustre-MDT0000 complete [ 1834.127755] LustreError: 64325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445937 with bad export cookie 7165273088344334009 [ 1834.134686] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1834.139535] LustreError: 64325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1834.427169] Lustre: server umount lustre-MDT0001 complete [ 1849.893022] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 1853.386367] Key type lgssc unregistered [ 1853.697782] LNet: 68109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1853.702195] LNetError: 68109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1853.715348] LNet: Removed LNI 192.168.204.150@tcp [ 1854.699844] Key type .llcrypt unregistered [ 1854.704413] Key type ._llcrypt unregistered [ 1878.747971] Key type ._llcrypt registered [ 1878.749766] Key type .llcrypt registered [ 1878.865587] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 1880.234926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1880.277080] alg: No test for adler32 (adler32-zlib) [ 1881.322275] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1881.580356] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 1883.271239] Key type lgssc registered [ 1884.393428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1892.420486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1892.454560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1894.054096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1899.080944] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1904.626570] LustreError: 69526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1904.661079] LustreError: 69526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1907.619567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1908.264528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1912.755839] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1920.111438] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1926.061850] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1932.450325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1933.073539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1938.715853] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1945.318887] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1956.514240] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1969.646555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1969.659917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1969.678594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1970.162777] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1970.195887] Lustre: Skipped 1 previous similar message [ 1970.207493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1970.223882] Lustre: Skipped 2 previous similar messages [ 1975.268204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1975.286088] Lustre: Skipped 2 previous similar messages [ 1976.005863] Lustre: server umount lustre-MDT0000 complete [ 1980.098634] LustreError: 69507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446083 with bad export cookie 12822794787195386781 [ 1980.112594] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1980.116434] LustreError: 69507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1980.388165] LustreError: 69521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1980.401233] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1980.455582] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1980.818473] Lustre: server umount lustre-MDT0001 complete [ 1986.965600] Lustre: server umount lustre-OST0000 complete [ 2015.190251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2015.650110] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2015.696577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2016.107413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2021.620945] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2032.680932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2032.796127] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2032.822121] Lustre: 74398:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2033.259385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2037.754723] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2044.275649] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2049.339831] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2056.149910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2056.303429] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2056.962450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2062.808206] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2071.460789] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2073.280390] Lustre: 76973:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2075.523539] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2085.349833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2085.357770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2085.365227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2085.381195] Lustre: Skipped 1 previous similar message [ 2088.017625] Lustre: server umount lustre-OST0000 complete [ 2094.560953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2094.565064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2094.582806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2094.588868] Lustre: Skipped 1 previous similar message [ 2106.335712] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2106.595829] Lustre: server umount lustre-MDT0000 complete [ 2110.349101] LustreError: 74391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446213 with bad export cookie 12822794787195387635 [ 2110.355595] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2110.365726] LustreError: 74391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2110.762673] Lustre: server umount lustre-MDT0001 complete [ 2122.085067] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 2125.765722] Key type lgssc unregistered [ 2126.189168] LNet: 78188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2126.199585] LNetError: 78188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2126.221089] LNet: Removed LNI 192.168.204.150@tcp [ 2127.475664] Key type .llcrypt unregistered [ 2127.477300] Key type ._llcrypt unregistered [ 2153.662243] Key type ._llcrypt registered [ 2153.664668] Key type .llcrypt registered [ 2153.770674] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 2154.828352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2155.019643] alg: No test for adler32 (adler32-zlib) [ 2156.224619] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2156.577548] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 2158.305363] Key type lgssc registered [ 2159.781662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2168.293857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2168.314455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2169.994873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2174.767094] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2179.559335] LustreError: 79604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2179.591526] LustreError: 79604:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2183.020695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2183.658450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2188.355641] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2196.125310] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2202.051257] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2208.689887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2209.231772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2215.540947] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2222.123709] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2232.198789] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2232.980386] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2234.789420] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2234.798921] Lustre: Skipped 1 previous similar message [ 2237.712366] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2237.717812] Lustre: Skipped 2 previous similar messages [ 2242.283758] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2242.289594] Lustre: Skipped 4 previous similar messages [ 2251.242459] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2251.247933] Lustre: Skipped 9 previous similar messages [ 2267.644140] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2267.650513] Lustre: Skipped 18 previous similar messages [ 2300.206191] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2300.216128] Lustre: Skipped 33 previous similar messages [ 2329.258884] Lustre: 87707:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2331.765302] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 10:40:33 (1788446433) [ 2333.183865] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2334.884235] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 10:40:36 (1788446436) [ 2341.347122] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2341.353440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2341.372541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2341.378532] Lustre: Skipped 1 previous similar message [ 2346.465291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2346.473891] Lustre: Skipped 1 previous similar message [ 2351.591771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2351.595357] Lustre: Skipped 1 previous similar message [ 2353.119788] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2353.275379] Lustre: server umount lustre-OST0000 complete [ 2357.733567] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2357.746780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2357.764965] Lustre: Skipped 1 previous similar message [ 2357.785344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2361.828771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2361.837204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2363.660452] Lustre: server umount lustre-MDT0000 complete [ 2366.949094] LustreError: 79600:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.965802] LustreError: 79600:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2367.859221] LustreError: 79586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446470 with bad export cookie 2368473924560566980 [ 2367.876073] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2367.883946] LustreError: 79586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2368.566304] Lustre: server umount lustre-MDT0001 complete [ 2380.487584] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 2383.477443] Key type lgssc unregistered [ 2383.845468] LNet: 88961:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2383.851925] LNetError: 88961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2383.871803] LNet: Removed LNI 192.168.204.150@tcp [ 2384.660477] Key type .llcrypt unregistered [ 2384.664839] Key type ._llcrypt unregistered [ 2388.750667] Key type ._llcrypt registered [ 2388.752652] Key type .llcrypt registered [ 2388.896230] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2393.051057] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules [ 2394.156794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2394.193713] alg: No test for adler32 (adler32-zlib) [ 2395.349934] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2395.632710] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 2397.327191] Key type lgssc registered [ 2398.919847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2400.088130] LustreError: 90165:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2417.112187] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 2421.543726] LustreError: 90165:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2425.944484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2425.972861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2427.569791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2432.439947] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2437.092517] LustreError: 91188:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2437.132541] LustreError: 91188:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2442.152795] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 10:42:23 (1788446543) [ 2442.214363] LustreError: 91189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.003680] LustreError: 91220:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2447.017681] LustreError: 91220:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2447.027562] LustreError: 91220:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2447.293593] Lustre: server umount lustre-MDT0000 complete [ 2459.993180] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 2462.928847] Key type lgssc unregistered [ 2463.233216] LNet: 92717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2463.245048] LNetError: 92717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2463.264693] LNet: Removed LNI 192.168.204.150@tcp [ 2464.170876] Key type .llcrypt unregistered [ 2464.178792] Key type ._llcrypt unregistered [ 2484.344100] Key type ._llcrypt registered [ 2484.348327] Key type .llcrypt registered [ 2484.534147] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 2485.652971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2485.670760] alg: No test for adler32 (adler32-zlib) [ 2486.955677] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2487.155820] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 2488.799237] Key type lgssc registered [ 2489.852811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2497.066051] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2497.090266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2498.626232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2502.447901] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2508.771450] LustreError: 93937:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.796813] LustreError: 93937:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2509.629363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2510.151434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2513.815549] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2519.972677] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2524.822171] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2531.122049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2531.605548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2537.301956] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2544.291839] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2591.766699] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 10:44:53 (1788446693) [ 2593.650559] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2595.614851] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 10:44:57 (1788446697) [ 2600.420597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2600.433854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2600.435403] Lustre: Skipped 1 previous similar message [ 2600.447427] Lustre: Skipped 2 previous similar messages [ 2605.539660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2605.547057] Lustre: Skipped 2 previous similar messages [ 2606.005823] Lustre: server umount lustre-MDT0000 complete [ 2610.289050] LustreError: 93917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446713 with bad export cookie 8751096536629258224 [ 2610.302224] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2610.658952] LustreError: 95965:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2610.670226] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2610.699822] Lustre: Skipped 1 previous similar message [ 2610.967112] Lustre: server umount lustre-MDT0001 complete [ 2617.715309] Lustre: server umount lustre-OST0000 complete [ 2628.415722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2628.913611] LustreError: 98212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2628.934137] LustreError: 98212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2629.107635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2633.306505] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2634.209093] LustreError: 98213:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2634.238887] LustreError: 98213:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2638.311110] LustreError: 98212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.364697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2642.185760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2647.059461] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2654.480933] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2660.070616] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2671.527543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2671.724857] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2671.733154] LustreError: 98199:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2671.767381] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2671.773600] LustreError: 100295:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2671.779300] LustreError: 100295:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2671.789043] LustreError: 100295:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2671.911465] Lustre: server umount lustre-OST0000 complete [ 2671.917962] LustreError: 100295:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2680.413613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2680.620803] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2680.632989] LustreError: 98202:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2680.639653] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2680.649628] LustreError: 100756:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2680.655531] LustreError: 100756:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2680.662785] LustreError: 100756:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2680.726290] Lustre: server umount lustre-OST0000 complete [ 2680.732792] LustreError: 100756:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2689.654266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2689.842962] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2689.846775] Lustre: Found index 0 for lustre-OST0000, updating log [ 2689.853567] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2689.930811] Lustre: lustre-OST0000: new disk, initializing [ 2689.934769] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2689.942253] Lustre: 101293:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2690.240766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2695.710557] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2699.230694] Lustre: Failing over lustre-OST0000 [ 2699.332343] Lustre: server umount lustre-OST0000 complete [ 2703.329704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2703.335044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2703.363175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2708.960789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2708.969715] Lustre: Skipped 2 previous similar messages [ 2713.127350] Lustre: server umount lustre-MDT0000 complete [ 2716.712235] LustreError: 98193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446819 with bad export cookie 8751096536629260968 [ 2716.716214] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2716.727503] LustreError: 98193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2717.107437] Lustre: server umount lustre-MDT0001 complete [ 2725.609208] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 10:47:07 (1788446827) [ 2741.540041] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 2751.629941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2752.065270] LustreError: 103642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2752.202295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2757.097325] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2765.033113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2769.539559] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2772.154488] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2777.832501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2778.485515] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2783.811819] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2790.766289] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2790.782755] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2790.800576] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2791.480093] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2791.553823] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2791.618779] Lustre: lustre-OST0001: new disk, initializing [ 2791.621817] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2791.626497] Lustre: 105960:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2791.892514] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:58 [ 2791.918270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2791.927694] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2791.932491] Lustre: Skipped 2 previous similar messages [ 2796.767316] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2797.026232] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 2797.034154] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2797.054597] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 2797.054996] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2797.061455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2800.191846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2800.205046] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2800.300860] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2806.105243] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2810.743215] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2815.971247] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2815.979729] Lustre: Skipped 2 previous similar messages [ 2815.982420] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2815.986811] Lustre: Skipped 4 previous similar messages [ 2820.546716] Lustre: server umount lustre-MDT0001 complete [ 2821.094659] LustreError: 105489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2821.117911] LustreError: 105489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2829.839744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2829.958871] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2829.967379] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2830.084570] Lustre: 103630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2831.924034] LustreError: 107555:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2831.929974] LustreError: 107555:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2831.936860] LustreError: 103634:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2831.988653] LustreError: 107566:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.150@tcp: cfg command failed: rc = -17 [ 2831.999397] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2831.999397] [ 2834.017245] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2835.954426] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2843.195988] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 10:49:04 (1788446944) [ 2860.053721] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 2876.777471] Lustre: 109579:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2895.986776] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2899.004883] Lustre: 110714:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2978.787850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2978.795103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2978.813489] Lustre: Skipped 1 previous similar message [ 2978.821575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2978.831161] Lustre: Skipped 1 previous similar message [ 2985.084397] Lustre: server umount lustre-MDT0000 complete [ 2987.866959] LustreError: 107458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447090 with bad export cookie 8751096536629261899 [ 2987.870848] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2987.878707] LustreError: 107458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2988.159951] Lustre: server umount lustre-MDT0001 complete [ 3001.979260] Lustre: server umount lustre-OST0000 complete [ 3005.728351] Lustre: 93246:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788447092/real 1788447092] req@ffff8a56037dd180 x1875322171896320/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788447108 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3005.766658] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3005.772743] Lustre: Skipped 3 previous similar messages [ 3006.559870] Lustre: server umount lustre-OST0001 complete [ 3012.098464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3021.544473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3029.680373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3038.965115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3058.576227] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 3069.389168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3069.401711] Lustre: lustre-MDT0000: reset Object Index mappings [ 3069.822376] LustreError: 114664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3069.830900] LustreError: 114664:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3069.936169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3069.943984] Lustre: Skipped 1 previous similar message [ 3075.069934] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3083.445890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3083.478712] Lustre: lustre-MDT0001: reset Object Index mappings [ 3089.570261] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3092.590741] Lustre: 115813:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3099.823304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3107.423409] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3108.946501] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3115.005441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3116.518050] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3122.687655] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3122.699288] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3124.006762] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3131.806722] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3135.233467] Lustre: 117689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3182.687945] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 10:54:44 (1788447284) [ 3209.702048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3209.707458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3209.727202] Lustre: Skipped 2 previous similar messages [ 3209.741851] Lustre: Skipped 9 previous similar messages [ 3214.828924] LustreError: 116901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3214.859545] LustreError: 116901:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 3214.903712] Lustre: server umount lustre-MDT0000 complete [ 3217.936447] LustreError: 114644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447320 with bad export cookie 8751096536629308995 [ 3217.963436] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3224.454369] Lustre: server umount lustre-MDT0001 complete [ 3234.194595] Lustre: server umount lustre-OST0000 complete [ 3243.705291] Lustre: server umount lustre-OST0001 complete [ 3250.466815] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 3259.833380] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 3291.443951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3291.838499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3291.880932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3291.973994] Lustre: lustre-MDT0000: new disk, initializing [ 3292.046848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3292.055600] Lustre: Skipped 3 previous similar messages [ 3292.080360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3296.328918] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3302.479349] Lustre: server umount lustre-MDT0000 complete [ 3311.058487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3311.290818] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3311.310280] Lustre: 121953:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3311.320983] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3315.541655] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3320.297889] Lustre: server umount lustre-MDT0000 complete [ 3328.810886] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 10:57:10 (1788447430) [ 3330.449804] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3332.507082] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 10:57:13 (1788447433) [ 3353.623694] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 3363.497828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3367.971724] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3376.992131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3377.102378] Lustre: 124002:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3377.151395] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3377.157748] Lustre: Skipped 1 previous similar message [ 3377.236911] Lustre: lustre-MDT0001: new disk, initializing [ 3377.385134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3377.396279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3382.314117] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3387.680952] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3393.817949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3394.083176] Lustre: lustre-OST0000: new disk, initializing [ 3394.086209] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3394.092055] Lustre: 125580:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3395.337655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3395.349847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3395.568308] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3401.001913] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3412.267207] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3412.452991] Lustre: lustre-OST0001: new disk, initializing [ 3412.458824] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3412.470579] Lustre: 126451:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3414.195259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3414.210784] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3414.341420] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3420.384592] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3430.492184] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3434.187224] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3798.343535] Lustre: 132872:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3798.349479] Lustre: 132872:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3798.353180] Lustre: 132872:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3798.358408] Lustre: 132872:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3798.369105] Lustre: 132872:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3798.375893] Lustre: 132872:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4095.598568] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:09:56 (1788448196) [ 4109.793879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4109.806103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4109.819504] Lustre: Skipped 3 previous similar messages [ 4109.825495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4109.829618] Lustre: Skipped 4 previous similar messages [ 4115.944295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4115.951969] Lustre: Skipped 7 previous similar messages [ 4116.193563] Lustre: server umount lustre-MDT0000 complete [ 4119.886346] LustreError: 132885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448222 with bad export cookie 8751096536629376657 [ 4119.888923] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4119.894153] LustreError: 132885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4121.057201] LustreError: 127289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4121.065633] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4121.075741] LustreError: 127289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4121.110902] Lustre: Skipped 3 previous similar messages [ 4124.069201] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4124.073949] Lustre: Skipped 2 previous similar messages [ 4134.372389] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4134.647296] Lustre: server umount lustre-MDT0001 complete [ 4153.311671] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4153.385240] Lustre: server umount lustre-OST0000 complete [ 4162.953965] Lustre: server umount lustre-OST0001 complete [ 4168.200584] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 4178.409786] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 4213.369905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4213.540344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4213.566220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4213.623825] Lustre: lustre-MDT0000: new disk, initializing [ 4213.672153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4213.676117] Lustre: Skipped 5 previous similar messages [ 4213.689979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4218.024581] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 4228.128885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4228.236613] Lustre: 135930:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4228.325160] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4228.335071] Lustre: Skipped 1 previous similar message [ 4228.434274] Lustre: lustre-MDT0001: new disk, initializing [ 4228.631524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4228.649390] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4233.484733] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 4242.688657] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4248.422476] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4255.330426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4255.561273] Lustre: lustre-OST0000: new disk, initializing [ 4255.566971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4255.573447] Lustre: 137898:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4255.657939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4255.660083] Lustre: Skipped 1 previous similar message [ 4256.829638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4256.839260] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4256.965771] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4261.834664] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 4270.789720] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4279.941756] Lustre: lustre-MDD0000: changelog on [ 4282.830250] Lustre: lustre-MDD0001: changelog on [ 6139.030435] Lustre: lustre-MDD0001: changelog off [ 6140.883714] Lustre: 138284:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6142.016763] Lustre: lustre-MDD0000: changelog off [ 6151.652867] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6151.654754] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6151.670983] Lustre: Skipped 1 previous similar message [ 6151.690370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6151.699233] Lustre: Skipped 5 previous similar messages [ 6154.454895] Lustre: server umount lustre-OST0000 complete [ 6158.816601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6158.827365] LustreError: Skipped 1 previous similar message [ 6158.836053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6158.859482] Lustre: Skipped 1 previous similar message [ 6158.866775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6161.893034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6164.328651] Lustre: server umount lustre-MDT0000 complete [ 6167.009083] LustreError: 135942:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6167.043813] LustreError: 135942:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6167.790802] LustreError: 135923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450270 with bad export cookie 8751096536630192885 [ 6167.802980] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6168.106226] Lustre: server umount lustre-MDT0001 complete [ 6180.527897] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 6183.339684] Key type lgssc unregistered [ 6183.575777] LNet: 147032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6183.583788] LNetError: 147032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6183.599084] LNet: Removed LNI 192.168.204.150@tcp [ 6184.267254] Key type .llcrypt unregistered [ 6184.268870] Key type ._llcrypt unregistered [ 6206.212538] Key type ._llcrypt registered [ 6206.215488] Key type .llcrypt registered [ 6206.319246] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 6222.254732] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 6223.702484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6223.732690] alg: No test for adler32 (adler32-zlib) [ 6224.857592] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6225.170079] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 6226.879824] Key type lgssc registered [ 6227.990517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6264.272456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6264.313630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6265.722875] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6265.800405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6265.949555] Lustre: lustre-MDT0000: new disk, initializing [ 6266.084354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6266.110745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6271.110421] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6281.768596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6281.917509] Lustre: 150231:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6281.965893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6281.969644] Lustre: Skipped 1 previous similar message [ 6282.032548] Lustre: lustre-MDT0001: new disk, initializing [ 6282.127771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6282.236398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6282.256958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6287.661064] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6298.183519] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6303.733539] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6310.072929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6310.323640] Lustre: lustre-OST0000: new disk, initializing [ 6310.327513] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6310.333588] Lustre: 152202:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6310.421758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6315.578589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6315.590723] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6315.766959] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6316.359414] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6324.463988] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6330.140804] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_module obdecho/obdecho [ 6331.825105] LustreError: 153207:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6331.833103] LustreError: 153207:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6333.631450] LustreError: 153304:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6346.722814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6346.727510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6346.738193] Lustre: Skipped 1 previous similar message [ 6346.763558] Lustre: Skipped 2 previous similar messages [ 6350.565967] Lustre: server umount lustre-MDT0000 complete [ 6354.211533] LustreError: 150223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450457 with bad export cookie 15998635075914857626 [ 6354.216932] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6354.226723] LustreError: 150223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6354.580728] Lustre: server umount lustre-MDT0001 complete [ 6369.022975] Lustre: server umount lustre-OST0000 complete [ 6392.284543] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 6405.223270] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 6440.914021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6441.130352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6441.161904] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6441.269160] Lustre: lustre-MDT0000: new disk, initializing [ 6441.333177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6441.347529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6445.868955] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6456.514214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6456.672656] Lustre: 157022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6456.744970] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6456.752656] Lustre: Skipped 1 previous similar message [ 6456.863905] Lustre: lustre-MDT0001: new disk, initializing [ 6456.939876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6456.980490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6457.003642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6462.078490] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6472.261739] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6478.075934] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6485.142932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6485.554946] Lustre: lustre-OST0000: new disk, initializing [ 6485.568842] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6485.581390] Lustre: 158989:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6485.717527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6491.204898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6491.213560] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6491.327473] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6492.069294] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6501.789799] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6507.652110] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6507.998918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6513.219248] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6513.399448] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6517.216776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6517.217176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6517.229539] LustreError: Skipped 1 previous similar message [ 6517.238126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6517.262148] Lustre: Skipped 2 previous similar messages [ 6521.880686] Lustre: server umount lustre-OST0000 complete [ 6527.456696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6527.487659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6527.496118] Lustre: Skipped 2 previous similar messages [ 6531.991894] Lustre: server umount lustre-MDT0000 complete [ 6532.575968] LustreError: 158583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6532.593609] LustreError: 158583:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6535.781516] LustreError: 157013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450638 with bad export cookie 15998635075914858585 [ 6535.784323] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6535.795372] LustreError: 157013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6536.257470] Lustre: server umount lustre-MDT0001 complete [ 6540.032555] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 11:50:41 (1788450641) [ 6541.737325] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6543.727451] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 11:50:45 (1788450645) [ 6551.217226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6551.676197] LustreError: 161403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6551.769545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6556.369295] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6557.153089] LustreError: 161404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6561.261365] LustreError: 161403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6565.652960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6571.366085] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6578.975809] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6584.071374] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6591.258677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6591.805586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6591.812312] Lustre: Skipped 1 previous similar message [ 6599.078925] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6606.688296] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6617.582142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6617.586312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6617.603569] Lustre: Skipped 3 previous similar messages [ 6617.612566] Lustre: Skipped 1 previous similar message [ 6619.320358] Lustre: server umount lustre-MDT0000 complete [ 6622.691897] LustreError: 161400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6622.713071] LustreError: 161400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6623.352658] LustreError: 161384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450726 with bad export cookie 15998635075914859467 [ 6623.358804] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6623.361122] LustreError: 161384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6627.822254] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6627.851753] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6627.862676] Lustre: Skipped 2 previous similar messages [ 6629.890418] Lustre: server umount lustre-MDT0001 complete [ 6637.130722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6651.623539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6651.887960] LustreError: 165247:0:(llog.c:1655:llog_backup()) MGC192.168.204.150@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6651.900271] Lustre: 165247:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.150@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6653.409139] LustreError: 165252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6653.422069] LustreError: 165252:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6673.889136] LustreError: 165252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6673.906962] LustreError: 165252:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6709.219983] LustreError: 163234:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6709.237494] LustreError: 163234:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [ 6712.799515] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6712.803674] Lustre: 165247:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.150@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6753.800696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6758.679660] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6758.900084] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6767.966544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6768.102424] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xde0691d64f029d01 [ 6768.113733] Lustre: MGC192.168.204.150@tcp: Connection restored to 0@lo (at 0@lo) [ 6773.070802] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6773.760959] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6780.766080] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6782.170847] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6787.691846] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6789.317171] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6809.578701] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6809.582532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6809.600304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6809.600313] Lustre: Skipped 2 previous similar messages [ 6809.612590] Lustre: Skipped 1 previous similar message [ 6812.880479] Lustre: server umount lustre-MDT0000 complete [ 6814.691865] LustreError: 163234:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6814.711544] LustreError: 163234:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 42 previous similar messages [ 6816.720548] LustreError: 161384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450919 with bad export cookie 15998635075914865921 [ 6816.721940] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6816.740436] LustreError: 161384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6817.166114] Lustre: server umount lustre-MDT0001 complete [ 6831.045185] Lustre: server umount lustre-OST0000 complete [ 6850.243960] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 6860.047440] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 6895.425518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6895.630631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6895.657443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6895.707971] Lustre: lustre-MDT0000: new disk, initializing [ 6895.761351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6895.765447] Lustre: Skipped 1 previous similar message [ 6895.776756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6900.871484] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6912.250329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6912.509590] Lustre: 170349:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6913.036168] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6918.458644] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6928.265237] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6934.256206] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6941.423258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6941.616486] Lustre: lustre-OST0000: new disk, initializing [ 6941.618216] Lustre: Skipped 1 previous similar message [ 6941.620090] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6941.622545] Lustre: Skipped 2 previous similar messages [ 6941.625848] Lustre: 172320:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6947.975386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6948.000021] Lustre: Skipped 1 previous similar message [ 6948.012708] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6948.116356] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6948.693925] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 6958.842963] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6963.845962] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6964.094949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6969.220968] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6969.463459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6973.924244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6973.929518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6973.936836] Lustre: Skipped 1 previous similar message [ 6973.938805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6973.974224] Lustre: Skipped 3 previous similar messages [ 6977.618535] Lustre: server umount lustre-OST0000 complete [ 6988.257241] Lustre: server umount lustre-MDT0000 complete [ 6989.290578] LustreError: 171210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6989.307740] LustreError: 171210:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6992.230311] LustreError: 170341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451095 with bad export cookie 15998635075914882756 [ 6992.235734] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6992.247180] LustreError: 170341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6992.704426] Lustre: server umount lustre-MDT0001 complete [ 7001.494970] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:58:23 (1788451103) [ 7008.811669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7014.051426] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7022.601651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7027.847716] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7034.480438] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7040.192774] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7047.105252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7053.367707] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7059.866931] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7069.538964] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:59:31 (1788451171) [ 7074.272706] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7074.279886] LustreError: Skipped 1 previous similar message [ 7074.282347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7074.298291] Lustre: Skipped 3 previous similar messages [ 7074.305908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7074.310374] Lustre: Skipped 2 previous similar messages [ 7079.905553] Lustre: server umount lustre-OST0000 complete [ 7084.512573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7084.519257] LustreError: Skipped 1 previous similar message [ 7090.000411] Lustre: server umount lustre-MDT0000 complete [ 7094.006696] LustreError: 174714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451197 with bad export cookie 15998635075914883631 [ 7094.030137] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7094.438561] Lustre: server umount lustre-MDT0001 complete [ 7104.606326] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 7107.424327] Key type lgssc unregistered [ 7107.807368] LNet: 178464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7107.826972] LNetError: 178464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7107.857275] LNet: Removed LNI 192.168.204.150@tcp [ 7108.815551] Key type .llcrypt unregistered [ 7108.818578] Key type ._llcrypt unregistered [ 7130.079754] Key type ._llcrypt registered [ 7130.081930] Key type .llcrypt registered [ 7130.206541] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 7131.525522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7131.883884] alg: No test for adler32 (adler32-zlib) [ 7132.921952] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7133.194080] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 7134.895339] Key type lgssc registered [ 7136.117745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7144.374752] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7144.412985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7160.031791] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7165.151795] LustreError: 179744:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.150@tcp: failed processing log, type 4: rc = -110 [ 7170.271557] LustreError: 179744:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7170.299426] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7170.305318] LustreError: MGC192.168.204.150@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7170.313363] LustreError: 179744:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7170.319076] LustreError: 179744:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7170.324040] LustreError: 179744:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7170.359592] Lustre: server umount lustre-OST0000 complete [ 7170.362594] LustreError: 179744:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7178.645554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7180.122935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7185.317531] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7190.500989] LustreError: 180253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7190.517563] LustreError: 180253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7194.428863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7194.774653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7199.617439] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7207.380687] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7213.704197] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7221.116979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7221.381546] LustreError: 182068:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7221.413305] Lustre: 182068:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.150@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7221.600504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7228.604420] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7236.437706] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7239.522669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7239.530474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7239.546839] Lustre: Skipped 2 previous similar messages [ 7239.560827] Lustre: Skipped 2 previous similar messages [ 7244.791643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7244.805066] LustreError: 180250:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7245.061959] Lustre: server umount lustre-MDT0000 complete [ 7249.314719] LustreError: 180235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451352 with bad export cookie 15283767687783408812 [ 7249.333448] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7249.337632] LustreError: 180235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7249.783267] LustreError: 180982:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000cbb5a63e x1875327043268608/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7249.914326] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7249.944192] LustreError: 180250:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7249.984814] LustreError: 180250:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7250.143229] Lustre: server umount lustre-MDT0001 complete [ 7256.504103] Lustre: server umount lustre-OST0000 complete [ 7266.719594] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 7278.148872] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 7315.795293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7316.115404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7316.156534] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7316.270746] Lustre: lustre-MDT0000: new disk, initializing [ 7316.390984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7316.402902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7320.445664] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7331.425482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7331.622710] Lustre: 185591:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7331.683644] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7331.705921] Lustre: Skipped 1 previous similar message [ 7331.785438] Lustre: lustre-MDT0001: new disk, initializing [ 7331.899324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7331.921941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7331.926951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7336.838269] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7347.022379] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7352.756915] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7360.541141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7360.761215] Lustre: lustre-OST0000: new disk, initializing [ 7360.765967] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7360.769989] Lustre: 187563:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7360.854519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7366.864545] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7369.781208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7369.798075] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7370.010975] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7376.485220] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7381.966403] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7382.203780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7387.018464] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7387.228873] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7390.687851] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7390.691468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7390.703108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7395.642804] Lustre: server umount lustre-OST0000 complete [ 7400.928679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7400.956805] Lustre: Skipped 1 previous similar message [ 7400.973806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7400.978040] Lustre: Skipped 2 previous similar messages [ 7405.906291] Lustre: server umount lustre-MDT0000 complete [ 7406.056867] LustreError: 186785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7406.074548] LustreError: 186785:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7410.148210] LustreError: 185604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7410.168944] LustreError: 185604:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7410.206615] LustreError: 185584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451513 with bad export cookie 15283767687783409533 [ 7410.207373] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7410.229712] LustreError: 185584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7410.684813] Lustre: server umount lustre-MDT0001 complete [ 7419.421911] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 12:05:21 (1788451521) [ 7439.869246] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 7443.126740] Key type lgssc unregistered [ 7443.471063] LNet: 190513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7443.482667] LNetError: 190513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7443.560716] LNet: Removed LNI 192.168.204.150@tcp [ 7444.439354] Key type .llcrypt unregistered [ 7444.445494] Key type ._llcrypt unregistered [ 7460.848697] Key type ._llcrypt registered [ 7460.851596] Key type .llcrypt registered [ 7460.994780] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 7462.215580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7462.280718] alg: No test for adler32 (adler32-zlib) [ 7463.316679] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7463.587638] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 7465.256446] Key type lgssc registered [ 7466.630911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7480.357916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7480.370678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7482.178257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7486.838679] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7492.579406] LustreError: 191733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7492.600895] LustreError: 191733:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7495.522864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7496.035728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7500.743992] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7508.156411] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7514.547770] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7522.859994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7523.055561] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7523.062679] LustreError: 193598:0:(llog.c:1696:llog_backup()) MGC192.168.204.150@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7523.075028] Lustre: 193598:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.150@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7523.202829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7530.130284] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7538.511565] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7540.711029] LustreError: lustre-MDT0000-lwp-MDT0001: operation fld_query to node 0@lo failed: rc = -107 [ 7540.721955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7540.737383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7542.246249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7542.254527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7542.274089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7545.830488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7545.837265] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7545.842139] Lustre: Skipped 1 previous similar message [ 7547.041989] Lustre: server umount lustre-MDT0000 complete [ 7550.950260] LustreError: 191733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7550.969289] LustreError: 191733:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7551.112200] LustreError: 191713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451654 with bad export cookie 14618295846718860596 [ 7551.116555] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7551.127487] LustreError: 191713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7552.991430] LustreError: 192464:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000005c16d99c x1875327389300224/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7553.242646] Lustre: server umount lustre-MDT0001 complete [ 7567.933546] Lustre: server umount lustre-OST0000 complete [ 7578.956536] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 7592.536372] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 7634.087674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7634.421159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7634.464783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7634.602084] Lustre: lustre-MDT0000: new disk, initializing [ 7634.744210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7634.801039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7639.984936] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7651.395736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7651.539451] Lustre: 197125:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7651.619536] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7651.622641] Lustre: Skipped 1 previous similar message [ 7651.626145] LustreError: 197909:0:(llog.c:1696:llog_backup()) MGC192.168.204.150@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7651.633384] LustreError: 197909:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7651.643692] Lustre: 197909:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.150@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7651.664876] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7651.670884] Lustre: Skipped 1 previous similar message [ 7651.906785] Lustre: lustre-MDT0001: new disk, initializing [ 7652.001525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7652.030566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7652.042935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7656.894393] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7668.205371] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7674.506171] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7682.285675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7682.551695] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7682.560304] Lustre: Skipped 1 previous similar message [ 7682.565440] LustreError: 199071:0:(llog.c:1696:llog_backup()) MGC192.168.204.150@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7682.573633] LustreError: 199071:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7682.587732] Lustre: 199071:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.150@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7682.596367] Lustre: 199071:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7682.614332] Lustre: lustre-OST0000: new disk, initializing [ 7682.618200] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7682.630140] Lustre: 199094:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7682.718205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7689.630968] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7691.351956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7691.364715] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7691.546859] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7699.991672] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7705.713478] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7705.939335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7710.672653] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7710.892941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7717.351724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7717.352603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7717.383166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7717.407421] Lustre: Skipped 2 previous similar messages [ 7718.886768] Lustre: server umount lustre-OST0000 complete [ 7724.017117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7724.031583] LustreError: Skipped 1 previous similar message [ 7724.040503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7724.051847] Lustre: Skipped 1 previous similar message [ 7724.064350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7729.573712] Lustre: server umount lustre-MDT0000 complete [ 7732.709063] LustreError: 197137:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7732.981391] LustreError: 197921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451836 with bad export cookie 14618295846718861317 [ 7732.988529] LustreError: 197921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7732.996229] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7733.410242] Lustre: server umount lustre-MDT0001 complete [ 7742.928950] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 12:10:44 (1788451844) [ 7761.214104] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 7772.211952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7772.781155] LustreError: 202232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7772.808324] LustreError: 202232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7772.852750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7777.486093] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7778.273164] LustreError: 202233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7786.324176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7792.080556] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7795.712210] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7802.265574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7809.448554] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7817.178966] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7817.364737] Lustre: lustre-OST0001: new disk, initializing [ 7817.372727] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7817.391789] Lustre: 204540:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7817.461078] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7817.469825] Lustre: Skipped 2 previous similar messages [ 7822.217701] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7822.907674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7822.915646] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7823.002497] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7833.024156] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7841.454881] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7844.263265] Lustre: DEBUG MARKER: ADD OST3 [ 7849.988416] loop: module loaded [ 7850.046694] loop0: detected capacity change from 0 to 204800000 [ 7850.195296] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7850.204474] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7850.216878] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7856.804444] loop0: detected capacity change from 0 to 204800000 [ 7857.454445] Lustre: DEBUG MARKER: STOP OST3 [ 7863.215971] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7863.517342] Lustre: lustre-OST0003: new disk, initializing [ 7863.529515] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7863.540648] Lustre: 206548:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7865.150484] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7865.305493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7865.321486] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7865.362157] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7867.168098] LustreError: 206701:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7867.268609] Lustre: Failing over lustre-OST0003 [ 7867.390059] Lustre: server umount lustre-OST0003 complete [ 7870.017020] LustreError: 203720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7870.025839] LustreError: 203720:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7870.441859] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7870.446913] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7870.457048] Lustre: Skipped 2 previous similar messages [ 7870.523714] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7878.133172] Lustre: DEBUG MARKER: START OST3 again [ 7884.322212] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7884.516251] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7884.523095] Lustre: Skipped 1 previous similar message [ 7884.542528] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7885.366620] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7885.737192] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7885.740064] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7893.908992] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7916.912993] Lustre: Failing over lustre-OST0003 [ 7916.981986] Lustre: server umount lustre-OST0003 complete [ 7920.622525] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7920.627977] LustreError: 203721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7920.635954] Lustre: Skipped 1 previous similar message [ 7920.656030] LustreError: 203721:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 7935.458291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7935.470346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7935.477421] Lustre: Skipped 2 previous similar messages [ 7940.290576] Lustre: server umount lustre-MDT0000 complete [ 7944.267253] LustreError: 202213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452047 with bad export cookie 14618295846718862199 [ 7944.269379] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7944.284574] LustreError: 202213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7950.857401] Lustre: server umount lustre-MDT0001 complete [ 7955.087632] Lustre: server umount lustre-OST0000 complete [ 7959.574444] Lustre: server umount lustre-OST0001 complete [ 7966.723979] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 7974.185368] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 8012.906220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8013.113960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8013.148754] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8013.199833] Lustre: lustre-MDT0000: new disk, initializing [ 8013.270446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8013.290924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8018.716551] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8029.692753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8029.871817] Lustre: 211998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8029.940536] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8029.946297] Lustre: Skipped 1 previous similar message [ 8030.022530] Lustre: lustre-MDT0001: new disk, initializing [ 8030.146263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8030.166345] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8034.491343] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8043.784085] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8050.133603] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8059.685572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8060.196125] Lustre: 213968:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8061.900578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8062.120832] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8068.688298] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8079.696821] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8084.164546] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8084.409120] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8090.193220] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8090.790766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8093.157559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8093.166957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8093.188193] Lustre: Skipped 6 previous similar messages [ 8093.200187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8093.208544] Lustre: Skipped 7 previous similar messages [ 8099.369779] Lustre: server umount lustre-OST0000 complete [ 8109.741625] Lustre: server umount lustre-MDT0000 complete [ 8113.124289] LustreError: 212008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8113.138718] LustreError: 212008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 8113.427819] LustreError: 212791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452216 with bad export cookie 14618295846718864495 [ 8113.429750] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8113.436621] LustreError: 212791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8113.868946] Lustre: server umount lustre-MDT0001 complete [ 8117.909628] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 12:16:59 (1788452219) [ 8134.997634] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 8145.999451] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 8182.501996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8182.837897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8182.872567] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8182.877457] Lustre: Skipped 1 previous similar message [ 8182.974901] Lustre: lustre-MDT0000: new disk, initializing [ 8182.978587] Lustre: Skipped 1 previous similar message [ 8183.067597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8183.073879] Lustre: Skipped 1 previous similar message [ 8187.483453] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8196.675356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8196.768052] Lustre: 218580:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8197.066161] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8200.676719] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8211.858857] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8218.795751] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8226.740719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8227.147898] Lustre: 220548:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8228.521083] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8234.078573] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8242.886724] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8247.946671] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8248.130800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8252.570311] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8252.783156] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8259.040350] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8259.048914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8259.049401] LustreError: Skipped 3 previous similar messages [ 8259.050516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8259.050522] Lustre: Skipped 5 previous similar messages [ 8259.101413] Lustre: Skipped 4 previous similar messages [ 8260.946870] Lustre: server umount lustre-OST0000 complete [ 8271.730441] Lustre: server umount lustre-MDT0000 complete [ 8274.401339] LustreError: 219458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8274.422950] LustreError: 219458:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8275.671602] LustreError: 218570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452378 with bad export cookie 14618295846718865377 [ 8275.678805] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8275.681785] LustreError: 218570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8276.128429] Lustre: server umount lustre-MDT0001 complete [ 8285.460587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8285.972588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8285.976514] Lustre: Skipped 5 previous similar messages [ 8290.531102] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8299.527793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8304.498960] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8312.987461] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8319.295694] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8325.727789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8332.572842] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8339.272899] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8349.669964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8349.677071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8349.683025] Lustre: Skipped 4 previous similar messages [ 8349.701777] LustreError: Skipped 1 previous similar message [ 8351.712841] Lustre: server umount lustre-OST0000 complete [ 8362.499228] Lustre: server umount lustre-MDT0000 complete [ 8366.329799] LustreError: 222743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452469 with bad export cookie 14618295846718866259 [ 8366.339404] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8366.352818] LustreError: 222743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8366.702395] Lustre: server umount lustre-MDT0001 complete [ 8480.970704] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 8484.380193] Key type lgssc unregistered [ 8484.679135] LNet: 226430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8484.687401] LNetError: 226430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8484.702918] LNet: Removed LNI 192.168.204.150@tcp [ 8485.862289] Key type .llcrypt unregistered [ 8485.868783] Key type ._llcrypt unregistered [ 8513.525651] Key type ._llcrypt registered [ 8513.528679] Key type .llcrypt registered [ 8513.664605] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 8531.050794] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 8532.008440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8532.017512] alg: No test for adler32 (adler32-zlib) [ 8533.284929] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8533.776120] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 8535.583639] Key type lgssc registered [ 8537.054518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8581.633903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8581.665788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8583.015843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8583.080568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8583.206357] Lustre: lustre-MDT0000: new disk, initializing [ 8583.340637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8583.372582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8588.704934] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8601.536555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8601.696464] Lustre: 229793:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8601.773689] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8601.776814] Lustre: Skipped 1 previous similar message [ 8601.831308] Lustre: lustre-MDT0001: new disk, initializing [ 8601.965702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8602.035898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8602.066425] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8607.739881] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8619.443841] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8625.272377] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8633.320738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8633.648540] Lustre: lustre-OST0000: new disk, initializing [ 8633.652722] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8633.669562] Lustre: 231761:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8633.802743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8640.072569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8640.087549] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8640.203255] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8641.222543] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8651.586170] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8656.974566] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8657.245627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8662.797295] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8663.067989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8666.080509] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8666.083128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8666.086755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8666.100862] Lustre: Skipped 1 previous similar message [ 8666.123141] Lustre: Skipped 1 previous similar message [ 8671.200518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8671.211756] Lustre: Skipped 1 previous similar message [ 8671.818541] Lustre: server umount lustre-OST0000 complete [ 8676.332850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8676.345478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8676.365271] Lustre: Skipped 1 previous similar message [ 8676.389308] Lustre: Skipped 1 previous similar message [ 8681.441280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8681.449323] Lustre: Skipped 1 previous similar message [ 8686.562580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8686.581466] Lustre: Skipped 1 previous similar message [ 8690.143141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8690.347899] Lustre: server umount lustre-MDT0000 complete [ 8694.821448] LustreError: 229785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452797 with bad export cookie 18066575013669586976 [ 8694.824867] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8694.835786] LustreError: 229785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8695.290503] Lustre: server umount lustre-MDT0001 complete [ 8714.718646] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 8727.755178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8728.221163] LustreError: 234699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8728.242529] LustreError: 234699:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8728.273915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8733.682963] LustreError: 234700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8733.954725] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8738.784953] LustreError: 234699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8742.609951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8743.053886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8747.586937] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8750.154955] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8756.717424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8757.051062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8763.377855] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8773.547506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8773.947038] Lustre: lustre-OST0001: new disk, initializing [ 8773.957218] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8773.972607] Lustre: 237004:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8774.117579] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8779.390727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8779.434840] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8779.641608] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8780.839564] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8793.925309] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8802.392403] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8816.299570] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 12:28:37 (1788452917) [ 8819.681522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8819.696724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8819.711436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8824.920326] Lustre: server umount lustre-MDT0000 complete [ 8825.827703] LustreError: 234700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8828.998392] LustreError: 234681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452932 with bad export cookie 18066575013669587858 [ 8829.001254] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8829.006625] LustreError: 234681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8829.431861] Lustre: server umount lustre-MDT0001 complete [ 8843.997849] Lustre: server umount lustre-OST0000 complete [ 8846.303398] Lustre: 228013:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788452933/real 1788452933] req@ffff8a571fea0700 x1875328511362432/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788452949 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8846.348499] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8846.385272] Lustre: Skipped 3 previous similar messages [ 8847.892599] Lustre: server umount lustre-OST0001 complete [ 8854.309887] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 8864.495744] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 8902.517299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8902.717455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8902.743853] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8902.825402] Lustre: lustre-MDT0000: new disk, initializing [ 8902.903084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8902.921160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8907.271688] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8917.210757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8917.339409] Lustre: 240950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8917.389786] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8917.400061] Lustre: Skipped 1 previous similar message [ 8917.470860] Lustre: lustre-MDT0001: new disk, initializing [ 8917.579882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8917.588447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8921.414624] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8930.337234] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8936.048987] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8942.882266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8943.159554] Lustre: lustre-OST0000: new disk, initializing [ 8943.162761] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8943.168367] Lustre: 242921:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8944.266711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8944.285464] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8944.384996] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8949.167204] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8962.146493] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8970.160593] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8970.839902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8976.707620] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8976.920198] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8980.454141] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8980.455653] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8980.461367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8980.469225] Lustre: Skipped 1 previous similar message [ 8980.476437] Lustre: Skipped 4 previous similar messages [ 8984.962260] Lustre: server umount lustre-OST0000 complete [ 8990.694593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8990.723056] Lustre: Skipped 1 previous similar message [ 8996.588521] Lustre: server umount lustre-MDT0000 complete [ 9000.497808] LustreError: 240944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453103 with bad export cookie 18066575013669590000 [ 9000.510257] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9001.113463] Lustre: server umount lustre-MDT0001 complete [ 9010.385402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9010.899983] LustreError: 245140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9010.925575] LustreError: 245140:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9011.004323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9011.012672] Lustre: Skipped 2 previous similar messages [ 9016.318576] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9020.393246] LustreError: 245140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9020.429444] LustreError: 245140:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9026.879634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9032.316064] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9039.849578] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9045.096933] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9052.097311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9059.419962] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9069.301877] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9219.556128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9219.567425] LustreError: Skipped 1 previous similar message [ 9219.574176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9219.591323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9219.606352] Lustre: Skipped 5 previous similar messages [ 9223.555588] Lustre: server umount lustre-OST0000 complete [ 9234.339108] Lustre: server umount lustre-MDT0000 complete [ 9235.936718] LustreError: 246262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9235.958330] LustreError: 246262:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9238.220540] LustreError: 245121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453341 with bad export cookie 18066575013669590875 [ 9238.228977] LustreError: 245121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9238.230837] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9238.816703] Lustre: server umount lustre-MDT0001 complete [ 9336.932726] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 9340.823290] Key type lgssc unregistered [ 9341.119474] LNet: 248758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9341.128294] LNetError: 248758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9341.143861] LNet: Removed LNI 192.168.204.150@tcp [ 9342.239345] Key type .llcrypt unregistered [ 9342.240724] Key type ._llcrypt unregistered [ 9373.542576] Key type ._llcrypt registered [ 9373.547413] Key type .llcrypt registered [ 9373.727137] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 9390.968142] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 9392.129928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9392.175172] alg: No test for adler32 (adler32-zlib) [ 9393.745108] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9394.221483] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 9395.967179] Key type lgssc registered [ 9397.571635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9452.501191] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 9463.687519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9463.731671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9465.019731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9465.045129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9465.128118] Lustre: lustre-MDT0000: new disk, initializing [ 9465.214932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9465.232369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9470.267688] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9481.362961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9481.461975] Lustre: 252833:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9481.572450] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9481.579446] Lustre: Skipped 1 previous similar message [ 9481.651468] Lustre: lustre-MDT0001: new disk, initializing [ 9481.749502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9481.794386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9481.816061] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9487.345893] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9492.730485] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9500.670144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9501.192426] Lustre: lustre-OST0000: new disk, initializing [ 9501.213455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9501.237242] Lustre: 254464:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9501.387244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9507.951123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9507.977605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9508.290034] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9512.113508] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9525.864588] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9526.058395] Lustre: lustre-OST0001: new disk, initializing [ 9526.063313] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9526.076743] Lustre: 255336:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9526.278031] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9531.772672] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9533.619172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9533.640613] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9533.763323] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9542.879811] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9550.636926] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9564.643245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9564.651823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9564.658625] Lustre: Skipped 3 previous similar messages [ 9564.665099] Lustre: Skipped 2 previous similar messages [ 9569.766990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9569.782455] Lustre: Skipped 4 previous similar messages [ 9570.178184] Lustre: server umount lustre-MDT0000 complete [ 9574.141159] LustreError: 252825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453677 with bad export cookie 454437033938342581 [ 9574.159332] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9574.164945] LustreError: 252825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9574.801193] Lustre: server umount lustre-MDT0001 complete [ 9588.084422] Lustre: server umount lustre-OST0000 complete [ 9590.244031] Lustre: 250340:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788453677/real 1788453677] req@ffff8a560bf00000 x1875329414150272/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788453693 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9590.297483] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9593.013740] Lustre: server umount lustre-OST0001 complete [ 9610.064411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9620.906237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9629.734651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9639.894500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9659.832780] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 9670.888603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9670.911065] Lustre: lustre-MDT0000: reset Object Index mappings [ 9671.491387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9676.340186] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9676.770939] LustreError: 260345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9676.789131] LustreError: 260345:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9681.893383] LustreError: 260344:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9684.960968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9684.980716] Lustre: lustre-MDT0001: reset Object Index mappings [ 9685.541664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9690.373045] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9693.283758] Lustre: 261486:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9699.198969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9706.271475] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9710.817936] LustreError: 261841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9714.688646] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9714.935709] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9714.939773] Lustre: Skipped 1 previous similar message [ 9717.014681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9717.017170] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9721.875844] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9729.862788] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9733.417613] Lustre: 263360:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9740.855533] Lustre: server umount lustre-MDT0000 complete [ 9741.798396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9741.805057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9741.826137] LustreError: 261077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9744.706397] LustreError: 260325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453847 with bad export cookie 454437033938349371 [ 9744.716757] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9744.728322] LustreError: 260325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9745.222363] Lustre: server umount lustre-MDT0001 complete [ 9759.721730] Lustre: server umount lustre-OST0000 complete [ 9763.687233] Lustre: 250337:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788453850/real 1788453850] req@ffff8a570301d180 x1875329414198016/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788453866 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9763.732248] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9763.745883] Lustre: Skipped 3 previous similar messages [ 9764.163422] Lustre: server umount lustre-OST0001 complete [ 9784.296989] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 12:44:45 (1788453885) [ 9802.367880] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 9814.625675] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 9854.575460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9854.869349] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9854.892982] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9854.963432] Lustre: lustre-MDT0000: new disk, initializing [ 9855.096612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9855.129867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9860.637771] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9872.850661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9872.970673] Lustre: 267615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9873.067198] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9873.072697] Lustre: Skipped 1 previous similar message [ 9873.171904] Lustre: lustre-MDT0001: new disk, initializing [ 9873.380933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9873.402574] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9879.178715] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9889.912306] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9896.160845] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9906.314742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9906.918718] Lustre: lustre-OST0000: new disk, initializing [ 9906.926437] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9906.937455] Lustre: 269588:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9907.995810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9908.032362] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9908.150952] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9915.237853] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9927.080537] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9933.155514] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9933.385083] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9938.411385] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9938.566820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9944.034553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9944.055095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9944.055243] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9946.728956] Lustre: server umount lustre-OST0000 complete [ 9954.273885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9954.280534] Lustre: Skipped 1 previous similar message [ 9954.288420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9954.299261] Lustre: Skipped 2 previous similar messages [ 9958.458815] Lustre: server umount lustre-MDT0000 complete [ 9959.405702] LustreError: 267624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9959.432405] LustreError: 267624:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9963.702701] LustreError: 267607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454066 with bad export cookie 454437033938351464 [ 9963.709901] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9963.726628] LustreError: 267607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9964.328584] Lustre: server umount lustre-MDT0001 complete [ 9989.373410] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [10001.313022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10001.806632] LustreError: 272559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10001.873708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10001.881898] Lustre: Skipped 2 previous similar messages [10007.031888] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10015.808075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10021.288084] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10024.592364] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10031.322740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10037.906037] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10047.832923] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10048.020724] Lustre: lustre-OST0001: new disk, initializing [10048.025150] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10048.032230] Lustre: 274870:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10049.524204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10049.542378] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10049.684171] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10054.921321] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10064.473408] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10068.842566] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10074.381375] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10074.707097] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10082.787438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10082.801124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10082.827924] Lustre: Skipped 1 previous similar message [10082.848667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10088.255133] Lustre: server umount lustre-MDT0000 complete [10090.467326] LustreError: 274055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10090.490309] LustreError: 274055:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10092.716143] LustreError: 272540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454195 with bad export cookie 454437033938352346 [10092.720543] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10092.735194] LustreError: 272540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10093.267774] Lustre: server umount lustre-MDT0001 complete [10107.225112] Lustre: server umount lustre-OST0000 complete [10121.590310] Lustre: server umount lustre-OST0001 complete [10140.079662] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [10152.632874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10153.120117] LustreError: 277927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10153.241147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10153.246522] Lustre: Skipped 3 previous similar messages [10158.111128] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10166.224850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10171.054472] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10174.270989] Lustre: 279070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10181.766344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10190.051399] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10198.051196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10200.006820] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10205.727366] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10207.270295] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10216.118892] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10219.853247] Lustre: 280947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10224.970478] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10225.038099] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10235.711765] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 12:52:17 (1788454337) [10251.750968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10251.762138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10251.789246] Lustre: Skipped 5 previous similar messages [10251.802137] Lustre: Skipped 7 previous similar messages [10253.100105] Lustre: server umount lustre-MDT0000 complete [10256.875664] LustreError: 277928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10256.916000] LustreError: 277928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [10257.094107] LustreError: 277908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454360 with bad export cookie 454437033938358898 [10257.097935] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10257.102224] LustreError: 277908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10263.685889] Lustre: server umount lustre-MDT0001 complete [10273.697385] Lustre: server umount lustre-OST0000 complete [10277.948668] Lustre: server umount lustre-OST0001 complete [10285.248098] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [10295.697894] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [10345.827928] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [10357.021874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10357.289620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10357.335356] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10357.441850] Lustre: lustre-MDT0000: new disk, initializing [10357.562034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10362.970581] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10372.770336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10372.850809] Lustre: 285187:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10373.137294] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10378.188282] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10391.667667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10392.185042] Lustre: 286820:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10394.102379] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10399.513172] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10409.552193] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10409.751606] Lustre: 287690:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10409.924256] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10409.941154] Lustre: Skipped 6 previous similar messages [10410.857143] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10410.879617] Lustre: Skipped 1 previous similar message [10411.042165] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10417.181906] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10429.338512] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10433.857077] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10433.861889] Lustre: Skipped 1 previous similar message [10449.279780] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 12:55:50 (1788454550) [10451.940045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10451.947519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10451.961266] Lustre: Skipped 6 previous similar messages [10451.969495] Lustre: Skipped 7 previous similar messages [10453.863985] Lustre: server umount lustre-MDT0000 complete [10457.067509] LustreError: 286074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10457.085847] LustreError: 286074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10457.404549] LustreError: 287689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454560 with bad export cookie 454437033938366542 [10457.415050] LustreError: 287689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10457.416295] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10457.632730] Lustre: server umount lustre-MDT0001 complete [10471.554476] Lustre: server umount lustre-OST0000 complete [10486.723876] Lustre: server umount lustre-OST0001 complete [10496.657882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10502.790944] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10512.564164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10512.865828] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10518.352301] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10526.063843] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10531.178830] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10538.287446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10547.178642] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10555.833495] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10574.117576] loop1: detected capacity change from 0 to 204800000 [10574.470164] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10574.499239] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10574.518120] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10580.468647] loop1: detected capacity change from 0 to 204800000 [10587.287236] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10587.396218] LustreError: 289924:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs951: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10587.406290] LustreError: 289924:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs951 [10587.410799] LustreError: 289924:0:(mgs_handler.c:517:mgs_target_reg()) fs951-MDT0001: Error checking target index: rc = -13 [10587.420307] LustreError: fs951-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10587.427720] LustreError: 293654:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10587.433758] LustreError: 293654:0:(tgt_mount.c:2129:server_put_super()) no obd fs951-MDT0001 [10587.439588] LustreError: 293654:0:(tgt_mount.c:116:server_deregister_mount()) fs951-MDT0001 not registered [10587.466214] Lustre: server umount fs951-MDT0001 complete [10587.468879] LustreError: 293654:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10597.234477] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10597.336290] Lustre: Setting parameter fs951-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs951-MDT0001 [10597.376203] Lustre: srv-fs951-MDT0001: No data found on store. Initialize space. [10597.379035] Lustre: Skipped 4 previous similar messages [10597.469728] Lustre: fs951-MDT0001: new disk, initializing [10597.477054] Lustre: Skipped 3 previous similar messages [10597.603473] Lustre: srv-fs951-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10597.631740] Lustre: 294122:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs951-MDT0001-mdtlov: can't create new llog: rc = -115 [10597.650308] LustreError: 294122:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs951-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10602.849896] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10609.917665] Lustre: server umount fs951-MDT0001 complete [10625.006369] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10628.067374] loop1: detected capacity change from 0 to 204800000 [10635.070900] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10635.681500] Lustre: 295689:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs951-MDT0001-mdtlov: can't create new llog: rc = -115 [10635.690629] LustreError: 295689:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs951-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10641.553415] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10647.160706] Lustre: server umount fs951-MDT0001 complete [10658.877268] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.50@tcp (stopping) [10660.835901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10660.864200] Lustre: Skipped 2 previous similar messages [10663.818072] Lustre: server umount lustre-MDT0000 complete [10674.293913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10674.491851] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10680.309828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10680.616665] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10689.622868] loop1: detected capacity change from 0 to 204800000 [10696.330487] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10696.816417] Lustre: 298140:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs951-MDT0001-mdtlov: can't create new llog: rc = -115 [10696.836978] LustreError: 298140:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs951-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10703.064337] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10708.375876] Lustre: server umount fs951-MDT0001 complete [10715.165890] LustreError: 291768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10715.172858] LustreError: 291768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 109 previous similar messages [10734.565342] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10734.587769] LustreError: Skipped 1 previous similar message [10739.643501] Lustre: server umount lustre-OST0000 complete [10746.851650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10746.866284] LustreError: Skipped 1 previous similar message [10750.473825] Lustre: server umount lustre-MDT0000 complete [10755.054899] LustreError: 250339:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8a573683b480 x1875329414577280/t0(0) o41->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 224/224 e 0 to 0 dl 1788454865 ref 1 fl Rpc:RESQU/200/0 rc -5/-107 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [10755.055896] LustreError: 289916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454858 with bad export cookie 454437033938369223 [10755.056991] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10755.125752] LustreError: 289916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10755.450821] Lustre: server umount lustre-MDT0001 complete [10765.525214] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [10768.039892] Key type lgssc unregistered [10768.515739] LNet: 300500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10768.527510] LNetError: 300500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10768.556845] LNet: Removed LNI 192.168.204.150@tcp [10769.290246] Key type .llcrypt unregistered [10769.295160] Key type ._llcrypt unregistered [10793.909687] Key type ._llcrypt registered [10793.916806] Key type .llcrypt registered [10794.084418] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [10795.614688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10795.667311] alg: No test for adler32 (adler32-zlib) [10796.642987] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10797.069108] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [10798.831200] Key type lgssc registered [10799.808484] Lustre: Echo OBD driver; http://www.lustre.org/ [10807.153737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10807.213035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10808.673423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10812.405828] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10818.038670] LustreError: 301775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10818.064706] LustreError: 301775:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10821.780305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10822.346123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10827.912931] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10836.218952] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10842.241844] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10849.066659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10849.572795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10853.682134] LustreError: 303605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10855.946187] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10858.991739] LustreError: 303850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10862.996980] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10864.111038] LustreError: 303604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10870.831064] LustreError: 303605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10870.869096] LustreError: 303605:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10881.065635] LustreError: 303605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10881.090271] LustreError: 303605:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10884.689908] loop1: detected capacity change from 0 to 204800000 [10884.956733] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10884.971234] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10884.982243] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10890.257567] loop1: detected capacity change from 0 to 204800000 [10897.351380] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10897.646649] Lustre: fs21926-OST0000: new disk, initializing [10897.651848] Lustre: srv-fs21926-OST0000: No data found on store. Initialize space. [10897.662547] Lustre: 305507:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10897.794621] Lustre: fs21926-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10897.897174] LustreError: 303606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10897.908952] LustreError: 303606:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [10904.742836] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10913.821585] Lustre: server umount fs21926-OST0000 complete [10924.029635] loop1: detected capacity change from 0 to 204800000 [10928.885159] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10929.062695] Lustre: fs21926-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10930.209528] LustreError: 303850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10930.243912] LustreError: 303850:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [10933.737812] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10934.900964] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10943.495829] loop2: detected capacity change from 0 to 204800000 [10943.734352] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10943.753092] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10943.769690] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10949.119152] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10949.614099] loop2: detected capacity change from 0 to 204800000 [10955.334030] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10955.411415] LustreError: fs21926-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10955.430707] LustreError: fs21926-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10955.445462] LustreError: 308548:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10955.451488] LustreError: 308548:0:(tgt_mount.c:2129:server_put_super()) no obd fs21926-OST0001 [10955.458348] LustreError: 308548:0:(tgt_mount.c:116:server_deregister_mount()) fs21926-OST0001 not registered [10955.490408] Lustre: server umount fs21926-OST0001 complete [10955.493815] LustreError: 308548:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10963.934547] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10964.129524] Lustre: fs21926-OST0001: new disk, initializing [10964.134915] Lustre: srv-fs21926-OST0001: No data found on store. Initialize space. [10964.141104] Lustre: 309100:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10964.220050] Lustre: fs21926-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10972.466239] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10983.006767] Lustre: server umount fs21926-OST0000 complete [10992.798672] Lustre: server umount fs21926-OST0001 complete [10995.169161] LustreError: 303850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10995.198111] LustreError: 303850:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 37 previous similar messages [11005.824384] loop1: detected capacity change from 0 to 204800000 [11011.531479] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11011.716348] Lustre: fs21926-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11018.447398] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11025.212562] loop2: detected capacity change from 0 to 204800000 [11030.950079] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11038.248886] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11047.061832] Lustre: server umount fs21926-OST0000 complete [11056.309957] Lustre: server umount fs21926-OST0001 complete [11068.897025] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11068.912918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11068.944138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11071.968102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11071.984661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11071.989663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11074.580664] Lustre: server umount lustre-OST0000 complete [11082.208559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11082.212425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11082.224556] Lustre: Skipped 1 previous similar message [11082.235544] Lustre: Skipped 2 previous similar messages [11085.549289] Lustre: server umount lustre-MDT0000 complete [11089.775413] LustreError: 301756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455192 with bad export cookie 11316087409929315915 [11089.801772] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11090.505887] Lustre: server umount lustre-MDT0001 complete [11102.053575] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [11105.722344] Key type lgssc unregistered [11106.064602] LNet: 315028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11106.078549] LNetError: 315028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11106.101382] LNet: Removed LNI 192.168.204.150@tcp [11107.481168] Key type .llcrypt unregistered [11107.487220] Key type ._llcrypt unregistered [11149.962725] Key type ._llcrypt registered [11149.964133] Key type .llcrypt registered [11150.231647] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [11168.615928] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [11169.641193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11169.719601] alg: No test for adler32 (adler32-zlib) [11171.015060] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11171.547550] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [11173.455230] Key type lgssc registered [11174.645490] Lustre: Echo OBD driver; http://www.lustre.org/ [11213.809110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11213.843065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11215.138849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11215.164921] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11215.275354] Lustre: lustre-MDT0000: new disk, initializing [11215.395824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11215.409291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11220.057326] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11231.520402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11231.709259] Lustre: 318892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11231.806966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11231.818516] Lustre: Skipped 1 previous similar message [11231.913752] Lustre: lustre-MDT0001: new disk, initializing [11232.067997] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11232.118995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11232.137043] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11237.078274] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11247.899844] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11254.507168] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11261.775053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11262.031355] Lustre: lustre-OST0000: new disk, initializing [11262.035726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11262.054616] Lustre: 320860:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11262.200816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11269.387817] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11269.779735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11269.799452] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11269.969477] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11281.208958] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11283.671116] Lustre: Mounted lustre-client - version 2.17.56_51_g2aa7cea [11284.926709] Lustre: Unmounted lustre-client [11295.727209] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11295.740034] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11295.764126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11299.330833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11299.341183] Lustre: Skipped 2 previous similar messages [11301.301963] Lustre: server umount lustre-OST0000 complete [11309.024348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11309.030419] LustreError: Skipped 1 previous similar message [11309.037517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11309.045910] Lustre: Skipped 1 previous similar message [11309.056295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11311.957916] Lustre: server umount lustre-MDT0000 complete [11315.286986] LustreError: 318884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455418 with bad export cookie 1748742527448964908 [11315.299781] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11315.309166] LustreError: 318884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11315.507256] Lustre: server umount lustre-MDT0001 complete [11326.101833] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [11330.119694] Key type lgssc unregistered [11340.511388] LNet: 1 local NIs in recovery (showing 1): 192.168.204.150@tcp [11340.768962] LNet: 323025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11340.787785] LNetError: 323025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11412.521070] LNet: Removed LNI 192.168.204.150@tcp [11413.449638] Key type .llcrypt unregistered [11413.452207] Key type ._llcrypt unregistered [11434.411561] Key type ._llcrypt registered [11434.419724] Key type .llcrypt registered [11434.698479] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [11454.264030] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [11455.713357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11455.731089] alg: No test for adler32 (adler32-zlib) [11456.984738] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11457.373239] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [11459.151424] Key type lgssc registered [11460.498184] Lustre: Echo OBD driver; http://www.lustre.org/ [11491.379445] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 13:13:13 (1788455593) [11498.271546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11498.300655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11499.652075] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11499.696238] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11499.810172] Lustre: lustre-MDT0000: new disk, initializing [11499.929950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11499.951180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11504.566601] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11516.173487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11516.363866] Lustre: 326267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11516.434642] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11516.439718] Lustre: Skipped 1 previous similar message [11516.571511] Lustre: lustre-MDT0001: new disk, initializing [11516.710836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11516.749142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11516.758149] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11522.149924] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11531.967188] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11537.857832] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11545.791144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11545.978338] Lustre: lustre-OST0000: new disk, initializing [11545.981951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11545.989063] Lustre: 328236:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11546.080387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11552.705680] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11555.516059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11555.538335] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11555.768660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11564.278347] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11581.412890] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11581.412890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11581.413972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11581.419621] Lustre: Skipped 1 previous similar message [11583.462596] Lustre: server umount lustre-OST0000 complete [11588.577874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11588.595826] LustreError: Skipped 1 previous similar message [11588.599223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11588.628119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11588.644096] Lustre: Skipped 1 previous similar message [11591.648795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11591.653857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11591.666481] Lustre: Skipped 1 previous similar message [11596.772075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11596.776160] Lustre: Skipped 1 previous similar message [11601.376385] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11601.641737] Lustre: server umount lustre-MDT0000 complete [11605.740543] LustreError: 326259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455708 with bad export cookie 3736944196286553560 [11605.745176] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11605.755873] LustreError: 326259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11606.229740] Lustre: server umount lustre-MDT0001 complete [11618.098942] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [11621.515479] Key type lgssc unregistered [11621.901090] LNet: 330147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11621.910800] LNetError: 330147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11621.932600] LNet: Removed LNI 192.168.204.150@tcp [11623.054693] Key type .llcrypt unregistered [11623.057771] Key type ._llcrypt unregistered [11645.771789] Key type ._llcrypt registered [11645.775665] Key type .llcrypt registered [11645.963096] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [11647.679778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11647.705661] alg: No test for adler32 (adler32-zlib) [11648.913604] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11649.161258] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [11650.801413] Key type lgssc registered [11652.428185] Lustre: Echo OBD driver; http://www.lustre.org/ [11661.859829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11661.872359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11663.396768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11668.029725] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11673.568563] LustreError: 331422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11673.588761] LustreError: 331422:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11675.316859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11675.595853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11679.304703] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11686.007284] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11691.605644] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11698.477733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11698.970904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11704.865910] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11711.228387] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11720.020932] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11721.161298] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11726.819765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11726.826987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11726.850040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11728.868456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11728.882480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11728.909463] Lustre: Skipped 1 previous similar message [11728.927667] Lustre: Skipped 2 previous similar messages [11731.350843] Lustre: server umount lustre-MDT0000 complete [11733.985754] LustreError: 331419:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11734.019801] LustreError: 331419:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11735.628842] LustreError: 331402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455838 with bad export cookie 6779456064022808855 [11735.630975] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11735.647654] LustreError: 331402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11739.105991] LustreError: 331417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11739.127621] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11739.139132] LustreError: 331417:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11739.169925] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11739.176784] Lustre: Skipped 1 previous similar message [11742.098593] Lustre: server umount lustre-MDT0001 complete [11747.003692] Lustre: server umount lustre-OST0000 complete [11757.644236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11763.013190] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11766.018583] Lustre: 335605:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.150@tcp [11768.042069] Lustre: server umount MGS complete [11777.078953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11777.524889] LustreError: 336176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11777.540790] LustreError: 336176:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11777.621309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11781.946589] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11782.624696] LustreError: 336177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11782.640741] LustreError: 336177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11789.703234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11790.133457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11794.518460] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11802.610049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11802.909347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11809.023419] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11815.847493] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11817.278301] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11822.267435] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11827.941857] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [11832.807270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11832.809441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11832.839059] Lustre: Skipped 1 previous similar message [11832.852662] Lustre: Skipped 3 previous similar messages [11834.488908] Lustre: server umount lustre-OST0000 complete [11841.507498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11841.529232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11841.549035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11844.888462] Lustre: server umount lustre-MDT0000 complete [11848.164200] LustreError: 336173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11848.185843] LustreError: 336173:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11848.820442] LustreError: 336158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455951 with bad export cookie 6779456064022809695 [11848.824281] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11848.831584] LustreError: 336158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11849.443546] Lustre: server umount lustre-MDT0001 complete [11860.838281] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [11863.710057] Key type lgssc unregistered [11864.056233] LNet: 339615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11864.061125] LNetError: 339615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11864.084901] LNet: Removed LNI 192.168.204.150@tcp [11864.904102] Key type .llcrypt unregistered [11864.905701] Key type ._llcrypt unregistered [11889.265846] Key type ._llcrypt registered [11889.271533] Key type .llcrypt registered [11889.450282] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [11906.896414] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [11907.876259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11907.896536] alg: No test for adler32 (adler32-zlib) [11908.986147] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11909.183534] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [11910.895387] Key type lgssc registered [11912.110877] Lustre: Echo OBD driver; http://www.lustre.org/ [11948.860813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11948.891031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11950.147356] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11950.170931] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11950.240560] Lustre: lustre-MDT0000: new disk, initializing [11950.322617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11950.345750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11954.172524] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11963.414979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11963.586757] Lustre: 342775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11963.657444] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11963.661123] Lustre: Skipped 1 previous similar message [11963.745815] Lustre: lustre-MDT0001: new disk, initializing [11963.846877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11963.892294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11963.910681] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11968.513176] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11976.985286] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11982.516848] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11988.969959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11989.274298] Lustre: lustre-OST0000: new disk, initializing [11989.281051] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11989.289434] Lustre: 344746:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11989.436089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11997.501630] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11998.848761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11998.882946] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11999.121203] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12009.760429] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12017.263980] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12017.479924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12022.643758] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12022.962139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12029.931664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12029.948708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12029.963277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12029.967414] Lustre: Skipped 1 previous similar message [12035.046246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12035.052971] Lustre: Skipped 1 previous similar message [12039.135153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12039.432977] Lustre: server umount lustre-OST0000 complete [12045.281725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12045.292892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12045.319351] Lustre: Skipped 2 previous similar messages [12050.056070] Lustre: server umount lustre-MDT0000 complete [12054.894817] LustreError: 343572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456157 with bad export cookie 3340857851307693635 [12054.911323] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12054.918788] LustreError: 343572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12055.258547] Lustre: server umount lustre-MDT0001 complete [12066.945645] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 13:22:48 (1788456168) [12086.064394] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [12098.340350] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [12136.281233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12136.634669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12136.690812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12136.801166] Lustre: lustre-MDT0000: new disk, initializing [12136.955597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12136.970878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12141.986796] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12153.560981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12153.689253] Lustre: 349508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12153.784365] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12153.790757] Lustre: Skipped 1 previous similar message [12153.934457] Lustre: lustre-MDT0001: new disk, initializing [12154.073246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12154.114860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12154.141791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12159.867334] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12169.222843] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12174.477849] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12181.782722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12182.208040] Lustre: lustre-OST0000: new disk, initializing [12182.214272] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12182.227655] Lustre: 351479:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12182.421743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12189.775600] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12189.809087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12189.828653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12190.057573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12200.500346] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12236.256761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12236.258911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12236.270037] LustreError: Skipped 1 previous similar message [12236.274561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12236.279119] Lustre: Skipped 1 previous similar message [12236.300412] Lustre: Skipped 2 previous similar messages [12241.396594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12241.403417] Lustre: Skipped 1 previous similar message [12242.271832] Lustre: server umount lustre-OST0000 complete [12251.616498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12251.617795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12251.646920] Lustre: Skipped 1 previous similar message [12251.674395] Lustre: Skipped 1 previous similar message [12254.192607] Lustre: server umount lustre-MDT0000 complete [12256.738117] LustreError: 349520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12256.779518] LustreError: 349520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12258.957139] LustreError: 349499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456361 with bad export cookie 3340857851307694517 [12258.976540] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12258.982441] LustreError: 349499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12259.561646] Lustre: server umount lustre-MDT0001 complete [12272.796617] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [12276.299155] Key type lgssc unregistered [12276.656925] LNet: 353659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12276.672476] LNetError: 353659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12277.734386] LNet: Removed LNI 192.168.204.150@tcp [12278.742088] Key type .llcrypt unregistered [12278.747044] Key type ._llcrypt unregistered [12301.966083] Key type ._llcrypt registered [12301.970666] Key type .llcrypt registered [12302.058690] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [12303.618995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12303.721204] alg: No test for adler32 (adler32-zlib) [12304.865466] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12305.158585] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [12306.879246] Key type lgssc registered [12308.393558] Lustre: Echo OBD driver; http://www.lustre.org/ [12318.195626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12318.207305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12319.931766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12325.508354] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12329.442085] LustreError: 354940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12329.463165] LustreError: 354940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12334.573722] LustreError: 354941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12335.336372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12335.875387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12340.412255] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12348.689781] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12354.872235] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12362.704985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12363.126147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12369.388306] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12376.280757] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12394.992604] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12395.012543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12395.057350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12396.002947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12396.007180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12396.017789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12396.069397] Lustre: Skipped 1 previous similar message [12398.518899] Lustre: server umount lustre-OST0000 complete [12406.245918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12406.265797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12411.363935] LustreError: 354940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12411.931048] Lustre: server umount lustre-MDT0000 complete [12416.488213] LustreError: 355668:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12416.525393] LustreError: 355668:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12417.183788] LustreError: 354921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456520 with bad export cookie 15413746539146970083 [12417.199608] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12417.205238] LustreError: 354921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12418.277292] Lustre: server umount lustre-MDT0001 complete [12428.007446] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [12432.087291] Key type lgssc unregistered [12432.595523] LNet: 360233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12432.611378] LNetError: 360233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12432.656260] LNet: Removed LNI 192.168.204.150@tcp [12434.117185] Key type .llcrypt unregistered [12434.120497] Key type ._llcrypt unregistered [12455.841049] Key type ._llcrypt registered [12455.847135] Key type .llcrypt registered [12456.003800] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [12457.423679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12457.606425] alg: No test for adler32 (adler32-zlib) [12458.827290] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12459.182028] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [12461.056093] Key type lgssc registered [12463.196534] Lustre: Echo OBD driver; http://www.lustre.org/ [12473.346626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12473.397725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12475.089277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12480.678110] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12485.601350] LustreError: 363722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12485.631663] LustreError: 363722:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12490.470739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12490.724694] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12490.922195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12497.101517] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12508.840853] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12515.587691] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12525.818246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12526.497941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12536.295844] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12545.818276] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12556.262346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12556.263348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12556.264276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12556.266098] LustreError: Skipped 1 previous similar message [12556.276602] Lustre: Skipped 1 previous similar message [12561.391163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12561.405041] Lustre: Skipped 2 previous similar messages [12562.903354] Lustre: server umount lustre-OST0000 complete [12571.618597] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12571.629458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12571.632393] Lustre: Skipped 1 previous similar message [12576.740146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12576.745082] Lustre: Skipped 2 previous similar messages [12582.371239] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12582.506521] Lustre: server umount lustre-MDT0000 complete [12586.982945] LustreError: 363722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12587.030028] LustreError: 363722:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12587.264821] LustreError: 363663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456690 with bad export cookie 9825274964177056311 [12587.298202] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12588.183109] Lustre: server umount lustre-MDT0001 complete [12602.482299] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [12606.739636] Key type lgssc unregistered [12607.293717] LNet: 371334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12607.314258] LNetError: 371334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12607.342154] LNet: Removed LNI 192.168.204.150@tcp [12608.391169] Key type .llcrypt unregistered [12608.393048] Key type ._llcrypt unregistered [12628.319189] Key type ._llcrypt registered [12628.322552] Key type .llcrypt registered [12628.588345] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [12629.929313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12629.962319] alg: No test for adler32 (adler32-zlib) [12631.125234] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12631.471154] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [12633.272086] Key type lgssc registered [12634.618861] Lustre: Echo OBD driver; http://www.lustre.org/ [12643.917482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12643.972657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12645.438403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12650.592936] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12655.635176] LustreError: 375194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12655.674881] LustreError: 375194:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12660.720239] LustreError: 375195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12661.365088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12661.803517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12666.217963] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12675.806828] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12684.726754] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12694.744570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12695.367399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12703.450928] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12713.745036] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12728.288556] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12728.297703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12728.314118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12731.360909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12731.366636] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12731.395223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12734.457487] Lustre: server umount lustre-OST0000 complete [12738.532253] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12738.547903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12738.567367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12738.571791] Lustre: Skipped 1 previous similar message [12741.601347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12741.605968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12741.615477] Lustre: Skipped 1 previous similar message [12746.721373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12746.734994] Lustre: Skipped 1 previous similar message [12752.351420] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12754.580460] Lustre: server umount lustre-MDT0000 complete [12756.964086] LustreError: 375186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12756.997654] LustreError: 375186:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12758.643243] LustreError: 375094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456861 with bad export cookie 9018801004534296231 [12758.644658] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12758.659716] LustreError: 375094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12759.271621] Lustre: server umount lustre-MDT0001 complete [12776.125300] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [12780.393857] Key type lgssc unregistered [12780.835749] LNet: 381032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12780.842084] LNetError: 381032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12780.859648] LNet: Removed LNI 192.168.204.150@tcp [12781.797992] Key type .llcrypt unregistered [12781.800147] Key type ._llcrypt unregistered [12890.437890] Key type ._llcrypt registered [12890.441142] Key type .llcrypt registered [12890.559240] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [12890.877401] Key type .llcrypt unregistered [12890.881602] Key type ._llcrypt unregistered [12973.846478] Key type ._llcrypt registered [12973.849574] Key type .llcrypt registered [12974.004103] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [12974.483424] Key type .llcrypt unregistered [12974.489934] Key type ._llcrypt unregistered [13008.559570] Key type ._llcrypt registered [13008.562315] Key type .llcrypt registered [13008.667196] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [13008.992539] Key type .llcrypt unregistered [13008.998224] Key type ._llcrypt unregistered [13041.029254] Key type ._llcrypt registered [13041.032775] Key type .llcrypt registered [13041.163903] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [13041.583149] Key type .llcrypt unregistered [13041.590617] Key type ._llcrypt unregistered [13127.711131] Key type ._llcrypt registered [13127.713335] Key type .llcrypt registered [13127.826079] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [13128.276651] Key type .llcrypt unregistered [13128.281967] Key type ._llcrypt unregistered [13159.290303] Key type ._llcrypt registered [13159.294348] Key type .llcrypt registered [13159.453864] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [13159.826536] Key type .llcrypt unregistered [13159.833471] Key type ._llcrypt unregistered [13192.102986] Key type ._llcrypt registered [13192.108624] Key type .llcrypt registered [13192.246824] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [13192.610358] Key type .llcrypt unregistered [13192.612889] Key type ._llcrypt unregistered [13223.508171] Key type ._llcrypt registered [13223.511691] Key type .llcrypt registered [13223.620038] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [13223.961334] Key type .llcrypt unregistered [13223.963768] Key type ._llcrypt unregistered [13259.389888] Key type ._llcrypt registered [13259.392953] Key type .llcrypt registered [13259.521541] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [13259.955477] Key type .llcrypt unregistered [13259.957534] Key type ._llcrypt unregistered [13292.948395] Key type ._llcrypt registered [13292.961246] Key type .llcrypt registered [13293.073115] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [13293.420468] Key type .llcrypt unregistered [13293.426126] Key type ._llcrypt unregistered [13361.512725] Key type ._llcrypt registered [13361.514752] Key type .llcrypt registered [13361.640063] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [13361.961428] Key type .llcrypt unregistered [13361.963093] Key type ._llcrypt unregistered [13395.287167] Key type ._llcrypt registered [13395.289824] Key type .llcrypt registered [13395.400277] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [13395.771392] Key type .llcrypt unregistered [13395.775225] Key type ._llcrypt unregistered [13423.085491] Key type ._llcrypt registered [13423.089745] Key type .llcrypt registered [13423.224544] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [13423.566322] Key type .llcrypt unregistered [13423.574457] Key type ._llcrypt unregistered [13481.425423] Key type ._llcrypt registered [13481.427427] Key type .llcrypt registered [13481.579509] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [13499.097571] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [13500.768410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13500.793293] alg: No test for adler32 (adler32-zlib) [13501.999028] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13502.228321] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [13503.892660] Key type lgssc registered [13505.404361] Lustre: Echo OBD driver; http://www.lustre.org/ [13542.929091] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13542.949863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13544.099470] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13544.158401] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13544.361938] Lustre: tst250fs-MDT0000: new disk, initializing [13544.471450] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13544.539823] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13548.657066] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13561.518465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13561.767313] Lustre: 401658:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13561.867392] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13561.873120] Lustre: Skipped 1 previous similar message [13561.960714] Lustre: tst250fs-MDT0001: new disk, initializing [13562.112823] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13562.172649] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13562.202226] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13567.958620] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13573.213153] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13581.276642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13581.503633] Lustre: tst250fs-OST0000: new disk, initializing [13581.508901] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13581.517721] Lustre: 403289:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13581.607897] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13585.611806] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13585.625427] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13585.739585] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13587.016756] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13599.769790] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13599.979176] Lustre: tst250fs-OST0001: new disk, initializing [13599.990768] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13600.005774] Lustre: 404161:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13600.186580] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13606.163719] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13606.180697] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13606.410244] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13609.458983] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13618.932603] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13621.289918] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13621.301263] Lustre: Skipped 1 previous similar message [13647.333795] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13647.355588] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13647.362527] Lustre: Skipped 3 previous similar messages [13647.366782] Lustre: Skipped 2 previous similar messages [13652.450971] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13652.462812] Lustre: Skipped 4 previous similar messages [13657.573242] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13659.103133] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13659.272795] Lustre: server umount tst250fs-MDT0000 complete [13662.844103] LustreError: 401647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457765 with bad export cookie 10239631910622213221 [13662.851390] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13662.857901] LustreError: 401647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13663.186128] Lustre: server umount tst250fs-MDT0001 complete [13677.372810] Lustre: server umount tst250fs-OST0000 complete [13691.968573] Lustre: server umount tst250fs-OST0001 complete [13714.408741] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [13722.538992] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [13764.865450] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 13:51:06 (1788457866) [13766.470141] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13785.311765] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [13796.421149] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [13834.220601] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13576 sec ========= 13:52:15 (1788457935) [13836.195359] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:52:17 (1788457937) === [13841.067775] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:52:22 (1788457942) === [13886.967630] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [13889.743467] Key type lgssc unregistered [13890.181628] LNet: 413298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13890.189933] LNetError: 413298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13890.211284] LNet: Removed LNI 192.168.204.150@tcp [13891.471171] Key type .llcrypt unregistered [13891.477206] Key type ._llcrypt unregistered