[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 459297367 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002411] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007449] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010123] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012032] SELinux: Initializing. [ 0.013074] *** VALIDATE selinux *** [ 0.022124] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026778] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028107] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029125] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030112] *** VALIDATE tmpfs *** [ 0.032363] *** VALIDATE proc *** [ 0.033239] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.036048] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038087] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040027] Spectre V2 : User space: Vulnerable [ 0.041006] Speculative Store Bypass: Vulnerable [ 0.043971] debug: unmapping init [mem 0xffffffff92259000-0xffffffff92260fff] [ 0.046158] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047577] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048025] ... version: 2 [ 0.049016] ... bit width: 48 [ 0.050013] ... generic registers: 4 [ 0.051015] ... value mask: 0000ffffffffffff [ 0.052017] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054015] ... event mask: 000000070000000f [ 0.055277] rcu: Hierarchical SRCU implementation. [ 0.057428] smp: Bringing up secondary CPUs ... [ 0.058641] x86: Booting SMP configuration: [ 0.059027] .... node #0, CPUs: #1 #2 #3 [ 0.063079] smp: Brought up 1 node, 4 CPUs [ 0.065014] smpboot: Max logical packages: 1 [ 0.066014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141875] node 0 deferred pages initialised in 73ms [ 0.143099] devtmpfs: initialized [ 0.144247] x86/mm: Memory block size: 128MB [ 0.147832] gcov: version magic: 0x41383552 [ 0.150177] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152071] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153279] pinctrl core: initialized pinctrl subsystem [ 0.154294] [ 0.155000] ************************************************************* [ 0.157017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160013] ** ** [ 0.163013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165011] ** ** [ 0.168012] ** This means that this kernel is built to expose internal ** [ 0.171016] ** IOMMU data structures, which may compromise security on ** [ 0.173012] ** your system. ** [ 0.175013] ** ** [ 0.178014] ** If you see this message and you are not debugging the ** [ 0.181013] ** kernel, report this immediately to your vendor! ** [ 0.184021] ** ** [ 0.186012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189013] ************************************************************* [ 0.192662] NET: Registered protocol family 16 [ 0.194561] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.196053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.198057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.201041] cpuidle: using governor menu [ 0.202764] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205439] PCI: Using configuration type 1 for base access [ 0.207225] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.217135] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.219040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.223135] cryptd: max_cpu_qlen set to 1000 [ 0.226250] ACPI: Added _OSI(Module Device) [ 0.228017] ACPI: Added _OSI(Processor Device) [ 0.230013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.232014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.237513] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.243538] ACPI: Interpreter enabled [ 0.244000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.247016] ACPI: Using IOAPIC for interrupt routing [ 0.249116] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.253513] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.264683] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.267040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.270019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.274090] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.280363] acpiphp: Slot [2] registered [ 0.282117] acpiphp: Slot [5] registered [ 0.283152] acpiphp: Slot [6] registered [ 0.285185] acpiphp: Slot [7] registered [ 0.287119] acpiphp: Slot [8] registered [ 0.288112] acpiphp: Slot [9] registered [ 0.290127] acpiphp: Slot [10] registered [ 0.292131] acpiphp: Slot [3] registered [ 0.293135] acpiphp: Slot [4] registered [ 0.295125] acpiphp: Slot [11] registered [ 0.297106] acpiphp: Slot [12] registered [ 0.299116] acpiphp: Slot [13] registered [ 0.301129] acpiphp: Slot [14] registered [ 0.302105] acpiphp: Slot [15] registered [ 0.303106] acpiphp: Slot [16] registered [ 0.305104] acpiphp: Slot [17] registered [ 0.306148] acpiphp: Slot [18] registered [ 0.308115] acpiphp: Slot [19] registered [ 0.310144] acpiphp: Slot [20] registered [ 0.312114] acpiphp: Slot [21] registered [ 0.314109] acpiphp: Slot [22] registered [ 0.315171] acpiphp: Slot [23] registered [ 0.317138] acpiphp: Slot [24] registered [ 0.319113] acpiphp: Slot [25] registered [ 0.321089] acpiphp: Slot [26] registered [ 0.322116] acpiphp: Slot [27] registered [ 0.324115] acpiphp: Slot [28] registered [ 0.326102] acpiphp: Slot [29] registered [ 0.327125] acpiphp: Slot [30] registered [ 0.329098] acpiphp: Slot [31] registered [ 0.330079] PCI host bridge to bus 0000:00 [ 0.332021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.334022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.337023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.340027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.342020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.345024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.346168] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.349953] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.352721] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.362013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.368014] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.371020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.374029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.376017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.380568] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.383936] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.387046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.390739] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.397014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.412017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.417790] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.424442] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.431013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.439015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.459015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.472598] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.481014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.489013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.507016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.521696] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.530014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.537015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.563014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.576726] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.584020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.596016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.622015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.635844] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.646015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.654073] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.680024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.696621] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.707015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.716015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.736016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.747666] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.753487] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.755342] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.757325] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.759167] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.763113] iommu: Default domain type: Passthrough [ 0.765469] SCSI subsystem initialized [ 0.767107] ACPI: bus type USB registered [ 0.768056] usbcore: registered new interface driver usbfs [ 0.770111] usbcore: registered new interface driver hub [ 0.772071] usbcore: registered new device driver usb [ 0.774192] pps_core: LinuxPPS API ver. 1 registered [ 0.777012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.780060] PTP clock support registered [ 0.783037] EDAC MC: Ver: 3.0.0 [ 0.785237] PCI: Using ACPI for IRQ routing [ 0.788091] NetLabel: Initializing [ 0.789020] NetLabel: domain hash size = 128 [ 0.790011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.793097] NetLabel: unlabeled traffic allowed by default [ 0.795146] vgaarb: loaded [ 0.797264] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.798010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.805435] clocksource: Switched to clocksource kvm-clock [ 0.908502] VFS: Disk quotas dquot_6.6.0 [ 0.910069] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.912312] *** VALIDATE ramfs *** [ 0.913187] *** VALIDATE hugetlbfs *** [ 0.914307] pnp: PnP ACPI init [ 0.916422] pnp: PnP ACPI: found 6 devices [ 0.937668] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.941383] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.943984] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.946605] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.949580] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.952441] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.955842] NET: Registered protocol family 2 [ 0.958682] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.964657] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.969231] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.974934] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.978452] TCP: Hash tables configured (established 65536 bind 65536) [ 0.982030] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.985688] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.989051] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.992389] NET: Registered protocol family 1 [ 0.995228] RPC: Registered named UNIX socket transport module. [ 0.998738] RPC: Registered udp transport module. [ 1.001751] RPC: Registered tcp transport module. [ 1.003515] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.006528] NET: Registered protocol family 44 [ 1.007933] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.010483] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.012666] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.015230] PCI: CLS 0 bytes, default 64 [ 1.017052] Unpacking initramfs... [ 2.431096] debug: unmapping init [mem 0xffff9c8c7cc54000-0xffff9c8c7ffbffff] [ 2.436483] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.439923] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.444603] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.931154] Initialise system trusted keyrings [ 2.932907] Key type blacklist registered [ 2.934291] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.942390] zbud: loaded [ 2.944528] *** VALIDATE nfs *** [ 2.945732] *** VALIDATE nfs4 *** [ 2.948646] pstore: using deflate compression [ 2.952895] Platform Keyring initialized [ 3.075093] NET: Registered protocol family 38 [ 3.077146] Key type asymmetric registered [ 3.078783] Asymmetric key parser 'x509' registered [ 3.080841] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.084161] io scheduler mq-deadline registered [ 3.086098] io scheduler kyber registered [ 3.087871] io scheduler bfq registered [ 3.090267] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.093557] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.095990] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.098672] ACPI: Power Button [PWRF] [ 3.104037] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.110416] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.131335] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.140107] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.160210] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.187807] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.219651] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.224925] Non-volatile memory driver v1.3 [ 3.226648] Linux agpgart interface v0.103 [ 3.260991] virtio_blk virtio1: [vda] 146008 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.264476] vda: detected capacity change from 0 to 74756096 [ 3.295674] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.299185] vdb: detected capacity change from 0 to 1073741824 [ 3.316642] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.319884] vdc: detected capacity change from 0 to 2621440000 [ 3.336779] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.340607] vdd: detected capacity change from 0 to 2621440000 [ 3.358593] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.362153] vde: detected capacity change from 0 to 4294967296 [ 3.382878] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.386523] vdf: detected capacity change from 0 to 4294967296 [ 3.395091] libphy: Fixed MDIO Bus: probed [ 3.400195] usbcore: registered new interface driver usbserial_generic [ 3.402922] usbserial: USB Serial support registered for generic [ 3.405169] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.410869] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.413216] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.417501] mousedev: PS/2 mouse device common for all mice [ 3.420242] rtc_cmos 00:05: RTC can wake from S4 [ 3.422951] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.423901] rtc_cmos 00:05: registered as rtc0 [ 3.433150] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.434736] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.436787] intel_pstate: CPU model not supported [ 3.442644] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.446595] hid: raw HID events driver (C) Jiri Kosina [ 3.449367] usbcore: registered new interface driver usbhid [ 3.451411] usbhid: USB HID core driver [ 3.453062] drop_monitor: Initializing network drop monitor service [ 3.455457] Initializing XFRM netlink socket [ 3.457373] NET: Registered protocol family 10 [ 3.460690] Segment Routing with IPv6 [ 3.462727] NET: Registered protocol family 17 [ 3.465206] mpls_gso: MPLS GSO support [ 3.471205] RAS: Correctable Errors collector initialized. [ 3.473500] AVX version of gcm_enc/dec engaged. [ 3.475161] AES CTR mode by8 optimization enabled [ 3.560409] sched_clock: Marking stable (3560374629, 0)->(4496438342, -936063713) [ 3.564457] registered taskstats version 1 [ 3.567167] Loading compiled-in X.509 certificates [ 3.569563] zswap: loaded using pool lzo/zbud [ 3.595880] Key type big_key registered [ 3.609852] Key type encrypted registered [ 3.611630] ima: No TPM chip found, activating TPM-bypass! [ 3.613788] ima: Allocated hash algorithm: sha1 [ 3.615502] ima: No architecture policies found [ 3.617202] evm: Initialising EVM extended attributes: [ 3.619454] evm: security.selinux [ 3.620898] evm: security.ima [ 3.622141] evm: security.capability [ 3.623412] evm: HMAC attrs: 0x1 [ 3.626274] rtc_cmos 00:05: setting system clock to 2026-08-19 04:54:17 UTC (1787115257) [ 3.633507] debug: unmapping init [mem 0xffffffff93203000-0xffffffff933fffff] [ 3.636670] debug: unmapping init [mem 0xffffffff91f82000-0xffffffff92258fff] [ 3.646069] Write protecting the kernel read-only data: 28672k [ 3.649420] debug: unmapping init [mem 0xffffffff90603000-0xffffffff907fffff] [ 3.651743] debug: unmapping init [mem 0xffffffff90f14000-0xffffffff90ffffff] [ 3.684421] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.693299] systemd[1]: Detected virtualization kvm. [ 3.695553] systemd[1]: Detected architecture x86-64. [ 3.697578] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.723392] systemd[1]: No hostname configured. [ 3.725232] systemd[1]: Set hostname to . [ 3.727251] random: systemd: uninitialized urandom read (16 bytes read) [ 3.729353] systemd[1]: Initializing machine ID from random generator. [ 3.774290] random: ln: uninitialized urandom read (6 bytes read) [ 3.883349] random: systemd: uninitialized urandom read (16 bytes read) [ 3.886059] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.890737] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.899381] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.465950] device-mapper: uevent: version 1.0.3 [ 4.468413] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System.[ 5.026508] random: fast init done [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.169719] virtio_net virtio0 ens2: renamed from eth0 [ 5.202778] scsi host0: ata_piix [ 5.209153] scsi host1: ata_piix [ 5.210946] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.213762] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.950803] dracut-initqueue[595]: RTNETLINK answers: File exists [ 10.129371] random: crng init done [ 10.131120] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.491799] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.638110] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.921042] SELinux: Disabled at runtime. [ 11.979755] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.989383] systemd[1]: Detected virtualization kvm. [ 11.990621] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.487765] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.491344] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.497339] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.501478] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.504598] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.515465] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.522500] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. Startin[ 12.614129] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS g Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.954691] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.333733] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.367241] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.466427] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.491335] EDAC sbridge: Ver: 1.1.2 [ 15.018698] Key type dns_resolver registered [ 15.338228] NFS: Registering the id_resolver key type [ 15.339752] Key type id_resolver registered [ 15.341050] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg350-server login: [ 41.215873] libcfs: loading out-of-tree module taints kernel. [ 41.237863] Key type ._llcrypt registered [ 41.239276] Key type .llcrypt registered [ 41.290577] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 49.956241] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 50.552526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.560959] alg: No test for adler32 (adler32-zlib) [ 51.580747] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 51.928615] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 53.567152] Key type lgssc registered [ 54.321765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 55.638011] hrtimer: interrupt took 3970852 ns [ 69.145922] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 117.157110] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 132.452210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 132.526451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 133.815342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 133.846642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 133.969349] Lustre: lustre-MDT0000: new disk, initializing [ 134.035285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 134.073944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 139.246472] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 158.417570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 158.809772] Lustre: 6521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 158.864266] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 158.874768] Lustre: Skipped 1 previous similar message [ 159.105552] Lustre: lustre-MDT0001: new disk, initializing [ 159.206390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 159.312751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 159.339278] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 165.854094] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.694935] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 181.988597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 182.402801] Lustre: lustre-OST0000: new disk, initializing [ 182.407194] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 182.416317] Lustre: 8458:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 182.538974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 188.975899] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.511858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 190.523890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 190.595331] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 203.908144] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 204.069782] Lustre: lustre-OST0001: new disk, initializing [ 204.078984] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 204.094497] Lustre: 9533:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 204.199639] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 210.498206] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.056140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 213.064946] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 213.107582] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 223.619899] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 231.458421] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 237.854993] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing check_logdir /tmp/testlogs/ [ 242.803858] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing yml_node [ 247.145843] Lustre: DEBUG MARKER: Client: 2.17.56.4 [ 249.538263] Lustre: DEBUG MARKER: MDS: 2.17.56.4 [ 251.840928] Lustre: DEBUG MARKER: OSS: 2.17.56.4 [ 253.481051] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 19 00:58:25 EDT 2026 [ 267.928904] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 269.252563] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 284.640152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 284.642718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 284.657551] Lustre: Skipped 1 previous similar message [ 284.672777] Lustre: Skipped 3 previous similar messages [ 289.763159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 289.783842] Lustre: Skipped 3 previous similar messages [ 290.297346] Lustre: server umount lustre-MDT0000 complete [ 293.382031] LustreError: 6512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787115547 with bad export cookie 15664341868108499312 [ 293.385028] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 293.395717] LustreError: 6512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 294.880343] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.891608] Lustre: Skipped 2 previous similar messages [ 294.895480] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 294.900062] Lustre: Skipped 1 previous similar message [ 298.912330] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 298.923242] Lustre: Skipped 1 previous similar message [ 299.714127] Lustre: server umount lustre-MDT0001 complete [ 303.293398] Lustre: server umount lustre-OST0000 complete [ 306.895805] Lustre: server umount lustre-OST0001 complete [ 313.932340] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 327.719180] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 357.896083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 358.170213] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 358.208271] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 358.287299] Lustre: lustre-MDT0000: new disk, initializing [ 358.414454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 358.443417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 363.555579] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 374.797853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 374.994486] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 375.124054] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 375.135370] Lustre: Skipped 1 previous similar message [ 375.247630] Lustre: lustre-MDT0001: new disk, initializing [ 375.392640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 375.434412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 375.452336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 379.784374] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 388.926362] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 396.372287] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 404.287748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 404.693430] Lustre: lustre-OST0000: new disk, initializing [ 404.701636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 404.709410] Lustre: 17051:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 404.786162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 406.767805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 406.780876] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 406.961979] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 411.733597] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 421.882213] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 426.569691] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 426.750803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 431.197558] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 431.400645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 437.728972] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 437.732258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 437.741650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 437.747289] LustreError: Skipped 1 previous similar message [ 437.777117] Lustre: Skipped 2 previous similar messages [ 439.288520] Lustre: server umount lustre-OST0000 complete [ 446.944697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 446.958565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.976301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 446.980071] Lustre: Skipped 1 previous similar message [ 449.439652] Lustre: server umount lustre-MDT0000 complete [ 453.088564] LustreError: 15094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 453.105488] LustreError: 15094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 453.459118] LustreError: 15887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787115707 with bad export cookie 15664341868108500845 [ 453.471932] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 453.481602] LustreError: 15887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 453.941840] Lustre: server umount lustre-MDT0001 complete [ 460.723683] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:01:52 (1787115712) [ 477.462802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 477.974855] LustreError: 19941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 478.047684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 482.228566] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 483.297628] LustreError: 19942:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 483.317687] LustreError: 19942:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 487.394328] LustreError: 19941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 489.809573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.548434] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 502.199896] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 507.557501] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 509.558480] Lustre: Failing over lustre-MDT0000 [ 509.807289] Lustre: server umount lustre-MDT0000 complete [ 510.947843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 510.962988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.985572] Lustre: Skipped 1 previous similar message [ 511.001770] LustreError: 19941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.161188] LustreError: 19941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.186427] LustreError: 19941:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 528.351327] Lustre: 3631:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787115766/real 1787115766] req@ffff9c8bc2182680 x1873926218214272/t0(0) o400->MGC192.168.203.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787115782 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 528.377332] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 528.592807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 538.596099] LustreError: 19942:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.618434] LustreError: 19942:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 543.326586] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 544.236475] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 544.238624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 544.273236] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 552.782290] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 558.566851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 558.571476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 558.582106] Lustre: Skipped 2 previous similar messages [ 558.594110] Lustre: Skipped 3 previous similar messages [ 562.747824] Lustre: server umount lustre-MDT0000 complete [ 565.944896] LustreError: 19921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787115819 with bad export cookie 15664341868108502259 [ 565.946853] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 565.957544] LustreError: 19921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 566.217647] Lustre: server umount lustre-MDT0001 complete [ 573.447367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 573.899700] LustreError: 23276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.914058] LustreError: 23276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 573.961116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 573.970857] Lustre: Skipped 2 previous similar messages [ 578.447555] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 586.184146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 590.415675] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 597.673477] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 603.175994] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 605.005573] Lustre: Failing over lustre-MDT0000 [ 605.226362] Lustre: server umount lustre-MDT0000 complete [ 607.199975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 607.207474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 623.684326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.844284] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 624.165154] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 628.282594] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 629.218943] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 629.231137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 629.237502] Lustre: Skipped 1 previous similar message [ 629.245761] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 635.156796] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 639.457257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 639.471195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 639.480549] Lustre: Skipped 1 previous similar message [ 639.487631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 639.492412] Lustre: Skipped 2 previous similar messages [ 653.279914] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 653.573938] Lustre: server umount lustre-MDT0000 complete [ 653.799885] LustreError: 23272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 653.814834] LustreError: 23272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 657.216484] LustreError: 23257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787115911 with bad export cookie 15664341868108503197 [ 657.224130] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 657.231044] LustreError: 23257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 657.710300] Lustre: server umount lustre-MDT0001 complete [ 665.831494] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:05:18 (1787115918) [ 680.428318] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 690.357414] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 726.787584] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 736.587365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 736.838817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 736.873692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 736.927566] Lustre: lustre-MDT0000: new disk, initializing [ 736.982715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 736.987226] Lustre: Skipped 2 previous similar messages [ 736.998330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 740.982937] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 749.956038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 750.071184] Lustre: 29648:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 750.308319] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 753.922276] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 762.011740] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 766.933844] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 773.155566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 773.380778] Lustre: 31616:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 775.225151] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 775.254321] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 775.297469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 775.316215] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 779.352885] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 787.825389] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 794.718565] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 811.489438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 811.496024] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 811.505499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 811.517190] Lustre: Skipped 2 previous similar messages [ 811.547887] Lustre: Skipped 5 previous similar messages [ 816.989414] Lustre: server umount lustre-OST0000 complete [ 826.330165] Lustre: server umount lustre-MDT0000 complete [ 826.851754] LustreError: 31745:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 826.878089] LustreError: 31745:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 829.444903] LustreError: 29641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116083 with bad export cookie 15664341868108503596 [ 829.452273] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 829.455417] LustreError: 29641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 829.735790] Lustre: server umount lustre-MDT0001 complete [ 839.218675] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 842.063974] Key type lgssc unregistered [ 842.354572] LNet: 33591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.367214] LNetError: 33591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 842.378710] LNet: Removed LNI 192.168.203.150@tcp [ 843.259253] Key type .llcrypt unregistered [ 843.263282] Key type ._llcrypt unregistered [ 864.753894] Key type ._llcrypt registered [ 864.755555] Key type .llcrypt registered [ 864.829967] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 865.850752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 865.873993] alg: No test for adler32 (adler32-zlib) [ 866.876697] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 867.151924] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 868.839183] Key type lgssc registered [ 869.968237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 876.820612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 876.837270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 878.318091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 882.670774] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 888.806856] LustreError: 35006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 888.828608] LustreError: 35006:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 890.560541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 890.919591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 894.639921] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 901.697186] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 906.099831] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 912.920395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 913.306194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 915.398855] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 919.098661] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 927.110743] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 939.410844] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:09:52 (1787116192) [ 941.252208] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 943.002773] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 948.942374] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:10:01 (1787116201) [ 956.673686] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:10:09 (1787116209) [ 957.571111] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 965.833994] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 965.838786] Lustre: Skipped 2 previous similar messages [ 967.442440] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:10:20 (1787116220) [ 971.112595] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 971.116381] Lustre: Skipped 3 previous similar messages [ 988.853893] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 988.860739] Lustre: Skipped 6 previous similar messages [ 991.547493] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:10:43 (1787116243) [ 994.368206] Lustre: *** cfs_fail_loc=131b, val=2*** [ 998.084172] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1006.368160] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:10:58 (1787116258) [ 1008.123629] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1008.133162] Lustre: Skipped 3 previous similar messages [ 1020.856502] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:11:13 (1787116273) [ 1036.267343] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1036.269075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1036.283169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1036.285820] Lustre: Skipped 1 previous similar message [ 1040.411988] Lustre: server umount lustre-OST0000 complete [ 1044.448242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1044.455707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1044.468161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1044.472513] Lustre: Skipped 1 previous similar message [ 1046.496843] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1046.498920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1046.523170] Lustre: Skipped 1 previous similar message [ 1050.039554] Lustre: server umount lustre-MDT0000 complete [ 1051.629459] LustreError: 35001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1051.663502] LustreError: 35001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1053.456069] LustreError: 34984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116307 with bad export cookie 3410683853091906274 [ 1053.458587] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1053.465345] LustreError: 34984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1053.789562] Lustre: server umount lustre-MDT0001 complete [ 1064.813793] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1067.397464] Key type lgssc unregistered [ 1067.712821] LNet: 42862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1067.724885] LNetError: 42862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1067.745237] LNet: Removed LNI 192.168.203.150@tcp [ 1068.590198] Key type .llcrypt unregistered [ 1068.592597] Key type ._llcrypt unregistered [ 1085.197428] Key type ._llcrypt registered [ 1085.201947] Key type .llcrypt registered [ 1085.370367] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 1100.719784] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1101.635090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1101.643553] alg: No test for adler32 (adler32-zlib) [ 1102.796982] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 1103.104594] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1104.839193] Key type lgssc registered [ 1105.865827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1137.569838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1137.607639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1138.856809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1138.912447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1138.999283] Lustre: lustre-MDT0000: new disk, initializing [ 1139.066108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1139.094274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1143.152294] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1152.528038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1152.742330] Lustre: 46020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1152.863881] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1152.873179] Lustre: Skipped 1 previous similar message [ 1153.015996] Lustre: lustre-MDT0001: new disk, initializing [ 1153.221463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1153.256711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1153.271430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1158.620491] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1170.135347] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1175.937459] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1182.538158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1182.887778] Lustre: lustre-OST0000: new disk, initializing [ 1182.896058] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1182.903921] Lustre: 47984:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1182.997720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1188.072399] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1190.469670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1190.484373] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1190.603491] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1196.319185] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1200.124300] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1200.293184] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1203.188067] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1203.352447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1206.252188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1206.255634] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1206.255758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1206.262480] Lustre: Skipped 1 previous similar message [ 1206.280742] LustreError: Skipped 1 previous similar message [ 1211.365461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1211.371012] Lustre: Skipped 1 previous similar message [ 1216.486798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1216.489610] Lustre: Skipped 1 previous similar message [ 1219.039231] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1219.356469] Lustre: server umount lustre-OST0000 complete [ 1225.184440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1225.192076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1225.226524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1226.748601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1231.356246] Lustre: server umount lustre-MDT0000 complete [ 1234.991337] LustreError: 46012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116488 with bad export cookie 16598864936744145341 [ 1235.004777] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1235.005586] LustreError: 46012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1235.217693] Lustre: server umount lustre-MDT0001 complete [ 1244.042821] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:14:56 (1787116496) [ 1250.014894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1250.395971] LustreError: 50446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1250.421908] LustreError: 50446:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1250.481617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1254.603441] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1255.907118] LustreError: 50447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1261.025134] LustreError: 50446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1262.190541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1262.482833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1266.711851] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1273.584077] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1278.621812] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1284.574653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1284.972862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1290.272897] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1297.580982] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1301.740339] LustreError: 52973:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1302.594681] Lustre: Modifying parameter general.timeout=1 in log params [ 1304.243333] Lustre: Modifying parameter general.timeout=3 in log params [ 1304.253151] Lustre: Skipped 1 previous similar message [ 1306.370165] Lustre: Modifying parameter general.timeout=6 in log params [ 1306.372926] Lustre: Skipped 2 previous similar messages [ 1310.969434] Lustre: Modifying parameter general.timeout=12 in log params [ 1310.974757] Lustre: Skipped 5 previous similar messages [ 1319.274889] Lustre: Modifying parameter general.timeout=23 in log params [ 1319.281769] Lustre: Skipped 10 previous similar messages [ 1336.005609] Lustre: Modifying parameter general.timeout=42 in log params [ 1336.020635] Lustre: Skipped 18 previous similar messages [ 1368.217888] Lustre: Modifying parameter general.timeout=75 in log params [ 1368.224771] Lustre: Skipped 32 previous similar messages [ 1427.098637] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:17:59 (1787116679) [ 1429.784919] Lustre: 59914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1431.842904] Lustre: 60025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1432.897518] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 1432.908154] Lustre: Skipped 58 previous similar messages [ 1433.878292] Lustre: 60125:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1438.094088] Lustre: 60323:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1438.106261] Lustre: 60323:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1464.569640] Lustre: 61793:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1464.577537] Lustre: 61793:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1464.610294] LustreError: 61793:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1468.474453] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:18:40 (1787116720) [ 1481.184554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1481.198631] Lustre: Skipped 2 previous similar messages [ 1481.200519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1481.210430] Lustre: Skipped 4 previous similar messages [ 1486.566723] Lustre: server umount lustre-MDT0000 complete [ 1490.429274] LustreError: 50428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116744 with bad export cookie 16598864936744146223 [ 1490.434517] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1490.443941] LustreError: 50428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1490.749585] Lustre: server umount lustre-MDT0001 complete [ 1505.303686] Lustre: server umount lustre-OST0000 complete [ 1534.212374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1534.543124] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1534.907092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1539.741210] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1549.460810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1549.575742] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1549.645643] Lustre: 64326:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1555.562204] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1562.493281] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1566.744865] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1573.278596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1573.475904] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1573.723955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1573.729364] Lustre: Skipped 1 previous similar message [ 1579.541848] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1586.584633] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1588.132662] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1588.145209] Lustre: Skipped 19 previous similar messages [ 1588.249315] Lustre: 66891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1596.897367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1596.909073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1596.930472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1596.936807] Lustre: Skipped 3 previous similar messages [ 1601.081375] Lustre: server umount lustre-OST0000 complete [ 1606.112451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1606.125725] LustreError: Skipped 1 previous similar message [ 1606.137202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1606.155966] Lustre: Skipped 1 previous similar message [ 1611.234986] Lustre: server umount lustre-MDT0000 complete [ 1612.769534] LustreError: 64338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1612.789090] LustreError: 64338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1614.527802] LustreError: 64318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116868 with bad export cookie 16598864936744149289 [ 1614.529829] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1614.537102] LustreError: 64318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1614.859124] Lustre: server umount lustre-MDT0001 complete [ 1625.893950] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1629.188617] Key type lgssc unregistered [ 1629.517258] LNet: 68102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1629.527567] LNetError: 68102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1629.551210] LNet: Removed LNI 192.168.203.150@tcp [ 1630.457221] Key type .llcrypt unregistered [ 1630.458885] Key type ._llcrypt unregistered [ 1654.146110] Key type ._llcrypt registered [ 1654.149964] Key type .llcrypt registered [ 1654.346974] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1655.711372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1655.719028] alg: No test for adler32 (adler32-zlib) [ 1656.915204] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 1657.288881] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1659.071196] Key type lgssc registered [ 1660.377864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1668.754647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1668.769669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1670.503364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1676.276536] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1680.870722] LustreError: 69523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1680.886872] LustreError: 69523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1685.860731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1685.986863] LustreError: 69524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1686.385606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1690.952300] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1698.580499] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1704.300354] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1710.044360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1710.571263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1716.418498] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1728.649919] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1738.219305] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1746.401689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.405608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1746.424117] Lustre: Skipped 1 previous similar message [ 1746.442669] Lustre: Skipped 2 previous similar messages [ 1751.520454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1751.527541] Lustre: Skipped 2 previous similar messages [ 1752.741835] Lustre: server umount lustre-MDT0000 complete [ 1756.644327] LustreError: 69520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1756.674066] LustreError: 69520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1756.832947] LustreError: 69504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117010 with bad export cookie 17242799931553476138 [ 1756.853853] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1757.177757] Lustre: server umount lustre-MDT0001 complete [ 1771.050646] Lustre: server umount lustre-OST0000 complete [ 1797.883840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1798.239390] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1798.301698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1798.592322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1803.394245] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1811.493800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.562535] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1811.582751] Lustre: 74400:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1811.887482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1816.042252] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1823.076898] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1828.451345] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1835.523075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1835.661702] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1835.994194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1841.887461] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1849.897682] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1851.437319] Lustre: 76973:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1853.416306] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1863.140360] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1863.162476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1863.183105] Lustre: Skipped 1 previous similar message [ 1863.191777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1863.660438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1865.132812] Lustre: server umount lustre-OST0000 complete [ 1873.376354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1873.391408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1873.414020] Lustre: Skipped 1 previous similar message [ 1873.419735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1873.428262] Lustre: Skipped 1 previous similar message [ 1879.011462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1879.024412] Lustre: Skipped 2 previous similar messages [ 1884.127362] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1884.447463] Lustre: server umount lustre-MDT0000 complete [ 1888.122545] LustreError: 75145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117141 with bad export cookie 17242799931553476978 [ 1888.128688] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1888.138905] LustreError: 75145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1888.461575] Lustre: server umount lustre-MDT0001 complete [ 1899.755604] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1902.245301] Key type lgssc unregistered [ 1902.566153] LNet: 78183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1902.578377] LNetError: 78183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1902.596874] LNet: Removed LNI 192.168.203.150@tcp [ 1903.574142] Key type .llcrypt unregistered [ 1903.575881] Key type ._llcrypt unregistered [ 1931.325578] Key type ._llcrypt registered [ 1931.328600] Key type .llcrypt registered [ 1931.452709] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1932.762590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1932.812314] alg: No test for adler32 (adler32-zlib) [ 1933.845959] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 1934.103746] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1935.791176] Key type lgssc registered [ 1936.991036] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1944.853311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1944.872085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1946.356471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1950.713703] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1956.838535] LustreError: 79597:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1956.858765] LustreError: 79597:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1959.154451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1959.640501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1964.080825] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1971.573285] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1976.967543] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1983.437415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1984.009471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1990.217762] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1996.947620] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2003.804515] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2004.944787] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2005.993062] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2008.007437] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2008.017171] Lustre: Skipped 1 previous similar message [ 2012.270741] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2012.280666] Lustre: Skipped 4 previous similar messages [ 2020.954666] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2020.958207] Lustre: Skipped 7 previous similar messages [ 2037.410744] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2037.426349] Lustre: Skipped 18 previous similar messages [ 2069.444416] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2069.452726] Lustre: Skipped 33 previous similar messages [ 2104.882162] Lustre: 87697:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2108.129174] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 01:29:20 (1787117360) [ 2109.968984] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2111.573955] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 01:29:24 (1787117364) [ 2119.139791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2119.146890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.152387] LustreError: Skipped 1 previous similar message [ 2119.161618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2122.121965] Lustre: server umount lustre-OST0000 complete [ 2128.352180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2128.378239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2128.395069] Lustre: Skipped 1 previous similar message [ 2128.402780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2128.408836] Lustre: Skipped 1 previous similar message [ 2129.378813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2129.422627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2129.430781] Lustre: Skipped 1 previous similar message [ 2132.076675] Lustre: server umount lustre-MDT0000 complete [ 2134.518486] LustreError: 80327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2134.546163] LustreError: 80327:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2136.233817] LustreError: 82185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117390 with bad export cookie 12253508194430876722 [ 2136.235569] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2136.252177] LustreError: 82185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2136.829741] Lustre: server umount lustre-MDT0001 complete [ 2151.023986] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2155.176794] Key type lgssc unregistered [ 2155.643764] LNet: 88949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2155.654576] LNetError: 88949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2155.680651] LNet: Removed LNI 192.168.203.150@tcp [ 2156.683224] Key type .llcrypt unregistered [ 2156.685147] Key type ._llcrypt unregistered [ 2162.036526] Key type ._llcrypt registered [ 2162.068578] Key type .llcrypt registered [ 2162.153684] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2166.377750] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules [ 2166.989525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2167.027323] alg: No test for adler32 (adler32-zlib) [ 2168.197850] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 2168.633447] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2170.384788] Key type lgssc registered [ 2171.814580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2173.219898] LustreError: 90152:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2190.228965] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2195.000143] LustreError: 90152:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2200.005432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2200.026082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2201.787772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2206.487834] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2211.297920] LustreError: 91177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2211.326193] LustreError: 91177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2214.728981] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 01:31:07 (1787117467) [ 2216.418723] LustreError: 91176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2219.047123] LustreError: 91208:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2219.061507] LustreError: 91208:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2219.072451] LustreError: 91208:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2219.406617] Lustre: server umount lustre-MDT0000 complete [ 2231.612122] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2234.514917] Key type lgssc unregistered [ 2234.871748] LNet: 92702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2234.886448] LNetError: 92702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2234.903599] LNet: Removed LNI 192.168.203.150@tcp [ 2235.664499] Key type .llcrypt unregistered [ 2235.672565] Key type ._llcrypt unregistered [ 2255.116833] Key type ._llcrypt registered [ 2255.121269] Key type .llcrypt registered [ 2255.219442] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2256.390518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2256.567628] alg: No test for adler32 (adler32-zlib) [ 2257.552170] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 2257.739270] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2259.439171] Key type lgssc registered [ 2260.624989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2270.572355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2270.584952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2272.337592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2278.521283] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2282.486664] LustreError: 93921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2282.508192] LustreError: 93921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2286.855714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2287.319880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2291.660545] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2298.868237] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2305.105924] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2311.666507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2312.410649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2318.836330] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2326.347674] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2377.123521] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 01:33:49 (1787117629) [ 2379.258591] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2381.556707] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 01:33:53 (1787117633) [ 2389.476849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2389.478992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2389.491966] Lustre: Skipped 2 previous similar messages [ 2389.492291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2389.506694] Lustre: Skipped 2 previous similar messages [ 2392.604776] Lustre: server umount lustre-MDT0000 complete [ 2394.595158] LustreError: 94652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2394.619445] LustreError: 94652:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2395.992620] LustreError: 94662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117649 with bad export cookie 15365075656545403088 [ 2396.000572] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2396.015883] LustreError: 94662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2396.411464] Lustre: server umount lustre-MDT0001 complete [ 2410.988205] Lustre: server umount lustre-OST0000 complete [ 2423.003504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2423.552619] LustreError: 98228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2423.725775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2428.617952] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2428.896760] LustreError: 98229:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2434.021714] LustreError: 98228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2439.207311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2439.773368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2445.548148] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2453.743322] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2459.216131] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2473.771066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2474.040504] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2474.045992] LustreError: 98217:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2474.062848] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2474.068860] LustreError: 100312:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2474.072788] LustreError: 100312:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2474.076964] LustreError: 100312:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2474.125424] Lustre: server umount lustre-OST0000 complete [ 2474.128318] LustreError: 100312:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2484.052960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2484.184777] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2484.198607] LustreError: 98216:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2484.228563] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2484.235885] LustreError: 100780:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2484.241874] LustreError: 100780:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2484.248277] LustreError: 100780:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2484.324361] Lustre: server umount lustre-OST0000 complete [ 2484.326455] LustreError: 100780:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2494.887636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2495.118915] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2495.124316] Lustre: Found index 0 for lustre-OST0000, updating log [ 2495.136938] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2495.267957] Lustre: lustre-OST0000: new disk, initializing [ 2495.282543] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2495.304115] Lustre: 101314:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2495.688164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2502.973654] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2507.410882] Lustre: Failing over lustre-OST0000 [ 2507.582609] Lustre: server umount lustre-OST0000 complete [ 2514.913346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2514.920596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2514.940905] Lustre: Skipped 1 previous similar message [ 2514.962443] Lustre: Skipped 1 previous similar message [ 2518.049964] Lustre: server umount lustre-MDT0000 complete [ 2520.033823] LustreError: 98229:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2520.049612] LustreError: 98229:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2522.137322] LustreError: 98209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117776 with bad export cookie 15365075656545405832 [ 2522.150509] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2522.158597] LustreError: 98209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2522.553446] Lustre: server umount lustre-MDT0001 complete [ 2531.247239] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 01:36:23 (1787117783) [ 2548.727311] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2559.510480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2560.065360] LustreError: 103661:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2560.215442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2565.417611] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2574.713444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2581.487040] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2585.537232] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2592.110490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2593.388277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2593.407527] Lustre: Skipped 1 previous similar message [ 2600.308752] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2609.057257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2609.186463] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2609.255762] Lustre: lustre-OST0001: new disk, initializing [ 2609.258203] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2609.264799] Lustre: 105977:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2610.269894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2610.282333] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2610.424662] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2616.402741] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2624.447383] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2631.924349] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2637.281097] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2637.284185] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.297493] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2639.705523] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.50@tcp (stopping) [ 2641.404114] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2641.418826] Lustre: Skipped 1 previous similar message [ 2642.822979] Lustre: server umount lustre-MDT0001 complete [ 2644.794172] LustreError: 103657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2644.809567] LustreError: 103657:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2653.205369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2653.361920] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2653.365767] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2653.406402] Lustre: 103650:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2654.938348] LustreError: 107530:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2654.947641] LustreError: 107530:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2654.964398] LustreError: 103653:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2655.131293] LustreError: 107550:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.150@tcp: cfg command failed: rc = -17 [ 2655.142934] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2655.142934] [ 2659.149322] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2659.329788] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2670.715320] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 01:38:42 (1787117922) [ 2690.514725] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2708.883764] Lustre: 109591:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2732.517000] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2736.245355] Lustre: 110726:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2822.625019] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2822.635622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2822.673455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2822.687918] Lustre: Skipped 3 previous similar messages [ 2825.091577] Lustre: server umount lustre-MDT0000 complete [ 2828.264964] LustreError: 103661:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2828.310773] LustreError: 103661:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 2831.185687] LustreError: 106696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118085 with bad export cookie 15365075656545406826 [ 2831.188704] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2831.195617] LustreError: 106696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2831.961548] Lustre: server umount lustre-MDT0001 complete [ 2847.835546] Lustre: server umount lustre-OST0000 complete [ 2849.695120] Lustre: 93232:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787118087/real 1787118087] req@ffff9c8cd034ea00 x1873928531572224/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787118103 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2849.722380] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2849.735632] Lustre: Skipped 3 previous similar messages [ 2852.093724] Lustre: server umount lustre-OST0001 complete [ 2858.458479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2868.799733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2878.033566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2888.741501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2910.451139] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2921.558782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2921.575314] Lustre: lustre-MDT0000: reset Object Index mappings [ 2922.684212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2922.691371] Lustre: Skipped 2 previous similar messages [ 2928.997306] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2940.371888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2940.423704] Lustre: lustre-MDT0001: reset Object Index mappings [ 2946.696667] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2950.295420] Lustre: 115836:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2957.623305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2965.759875] LustreError: 116193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2965.776107] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2965.780621] LustreError: 116193:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2966.006934] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2967.872323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 2976.014829] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2981.891989] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2981.895076] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2983.621500] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2991.605462] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2995.436500] Lustre: 117712:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3045.870923] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 01:44:58 (1787118298) [ 3074.017750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3074.021928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3074.075237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3074.098277] Lustre: Skipped 7 previous similar messages [ 3079.957426] Lustre: server umount lustre-MDT0000 complete [ 3083.664451] LustreError: 114668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118337 with bad export cookie 15365075656545453677 [ 3083.671491] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3083.675566] LustreError: 114668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3090.205266] Lustre: server umount lustre-MDT0001 complete [ 3093.952391] LustreError: 118476:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000400:0x79:0x0].0x0 (ffff9c8cfcfe3f00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3108.319120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3108.719703] Lustre: server umount lustre-OST0000 complete [ 3118.947366] Lustre: server umount lustre-OST0001 complete [ 3126.132519] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 3138.366464] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3180.677229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3181.064828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3181.111667] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3181.301387] Lustre: lustre-MDT0000: new disk, initializing [ 3181.441129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3181.444054] Lustre: Skipped 3 previous similar messages [ 3181.493791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3186.985512] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3194.698865] Lustre: server umount lustre-MDT0000 complete [ 3204.285838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3204.628588] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3204.645875] Lustre: 121987:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3204.660912] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3209.903470] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3214.912820] Lustre: server umount lustre-MDT0000 complete [ 3223.285956] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 01:47:55 (1787118475) [ 3225.068933] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3226.775843] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 01:47:59 (1787118479) [ 3243.641992] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3253.825628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3259.668690] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3268.894933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3269.084600] Lustre: 124034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3269.125925] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3269.129563] Lustre: Skipped 1 previous similar message [ 3269.192217] Lustre: lustre-MDT0001: new disk, initializing [ 3269.389970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3269.411940] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3273.840936] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3278.385837] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3284.773062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3285.081496] Lustre: lustre-OST0000: new disk, initializing [ 3285.089634] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3285.097429] Lustre: 125614:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3287.103863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3287.116568] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3287.297213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3291.615633] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3302.569435] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3302.814175] Lustre: lustre-OST0001: new disk, initializing [ 3302.834195] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3302.851704] Lustre: 126484:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3304.786991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3304.797570] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3304.991056] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3310.672890] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3321.154483] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3324.877942] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3721.416876] Lustre: 132901:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3721.425434] Lustre: 132901:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3721.430727] Lustre: 132901:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3721.438550] Lustre: 132901:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3721.452382] Lustre: 132901:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3721.464118] Lustre: 132901:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3988.125732] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:00:39 (1787119239) [ 4006.376360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4006.384935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4006.399698] Lustre: Skipped 6 previous similar messages [ 4006.414728] Lustre: Skipped 11 previous similar messages [ 4010.774712] Lustre: server umount lustre-MDT0000 complete [ 4011.493670] LustreError: 127296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4011.515815] LustreError: 127296:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 4014.644279] LustreError: 124028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119268 with bad export cookie 15365075656545521766 [ 4014.653859] LustreError: 124028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4014.661299] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4016.610667] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4016.617302] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4016.644032] Lustre: Skipped 3 previous similar messages [ 4016.668886] Lustre: Skipped 1 previous similar message [ 4021.215474] Lustre: server umount lustre-MDT0001 complete [ 4031.574731] Lustre: server umount lustre-OST0000 complete [ 4041.892318] Lustre: server umount lustre-OST0001 complete [ 4048.742732] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 4059.461094] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4098.312388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4098.700354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4098.770304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4098.897854] Lustre: lustre-MDT0000: new disk, initializing [ 4099.037840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4099.052742] Lustre: Skipped 5 previous similar messages [ 4099.067278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4105.187055] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4115.320624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4115.425059] Lustre: 135960:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4115.499124] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4115.501932] Lustre: Skipped 1 previous similar message [ 4115.582096] Lustre: lustre-MDT0001: new disk, initializing [ 4115.785811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4115.800617] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4120.050614] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4129.834853] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4135.414897] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4142.300462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4142.618153] Lustre: lustre-OST0000: new disk, initializing [ 4142.622329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4142.630174] Lustre: 137925:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4142.764519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4142.774519] Lustre: Skipped 1 previous similar message [ 4143.752367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4143.760972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4143.896141] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4149.550948] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4159.425637] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4169.275526] Lustre: lustre-MDD0000: changelog on [ 4171.897694] Lustre: lustre-MDD0001: changelog on [ 5929.482958] Lustre: lustre-MDD0001: changelog off [ 5931.232299] Lustre: 137984:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5932.120616] Lustre: lustre-MDD0000: changelog off [ 5941.220066] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5941.222689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5941.224717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5941.224724] Lustre: Skipped 2 previous similar messages [ 5941.235868] LustreError: Skipped 1 previous similar message [ 5941.279149] Lustre: Skipped 1 previous similar message [ 5944.832433] Lustre: server umount lustre-OST0000 complete [ 5948.897512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5948.904783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5948.927935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5948.937887] Lustre: Skipped 1 previous similar message [ 5951.459854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5951.461053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5955.313675] Lustre: server umount lustre-MDT0000 complete [ 5956.591818] LustreError: 139731:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5956.611706] LustreError: 139731:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5958.876825] LustreError: 135952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121212 with bad export cookie 15365075656546299760 [ 5958.886652] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5958.902532] LustreError: 135952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5959.558247] Lustre: server umount lustre-MDT0001 complete [ 5969.672842] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5972.709553] Key type lgssc unregistered [ 5973.037943] LNet: 147034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5973.045085] LNetError: 147034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5973.059874] LNet: Removed LNI 192.168.203.150@tcp [ 5974.057135] Key type .llcrypt unregistered [ 5974.061689] Key type ._llcrypt unregistered [ 5995.307264] Key type ._llcrypt registered [ 5995.309888] Key type .llcrypt registered [ 5995.435683] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 6010.280567] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 6011.653149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6011.681072] alg: No test for adler32 (adler32-zlib) [ 6012.682951] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 6012.937028] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 6014.584885] Key type lgssc registered [ 6015.814366] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6048.362630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6048.388884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6049.695540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6049.729952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6049.794872] Lustre: lustre-MDT0000: new disk, initializing [ 6049.907871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6049.920641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6053.725185] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6064.426531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6064.582636] Lustre: 150233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6064.663301] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6064.674711] Lustre: Skipped 1 previous similar message [ 6064.740684] Lustre: lustre-MDT0001: new disk, initializing [ 6064.887889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6064.950800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6064.965791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6070.219827] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6080.841696] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6085.951852] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6092.112620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6092.343279] Lustre: lustre-OST0000: new disk, initializing [ 6092.347371] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6092.353504] Lustre: 152202:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6092.458964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6096.886231] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6100.017505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6100.027719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6100.142239] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6105.273437] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6110.834615] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_module obdecho/obdecho [ 6112.400966] LustreError: 153204:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6112.407968] LustreError: 153204:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6114.149404] LustreError: 153300:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6126.048925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6126.055205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6126.069085] Lustre: Skipped 1 previous similar message [ 6126.076945] Lustre: Skipped 2 previous similar messages [ 6131.119284] Lustre: server umount lustre-MDT0000 complete [ 6134.749516] LustreError: 152201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121388 with bad export cookie 14229064300469719564 [ 6134.756821] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6134.763087] LustreError: 152201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6135.055490] Lustre: server umount lustre-MDT0001 complete [ 6149.243188] Lustre: server umount lustre-OST0000 complete [ 6170.036639] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 6181.881771] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 6219.268546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6219.579961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6219.599810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6219.676852] Lustre: lustre-MDT0000: new disk, initializing [ 6219.795959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6219.826141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6224.399719] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6234.451046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6234.599916] Lustre: 157018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6234.664342] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6234.668531] Lustre: Skipped 1 previous similar message [ 6234.747641] Lustre: lustre-MDT0001: new disk, initializing [ 6234.850976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6234.887147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6234.905156] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6238.909693] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6247.945414] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6253.254902] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6259.533508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6259.797137] Lustre: lustre-OST0000: new disk, initializing [ 6259.804596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6259.813240] Lustre: 158984:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6259.913737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6266.092455] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6268.490203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6268.500484] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6268.585444] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6274.892438] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6279.527758] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6279.805481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6284.700087] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6284.981873] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6289.379832] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6289.380809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6289.398984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6289.426865] Lustre: Skipped 2 previous similar messages [ 6289.451980] Lustre: Skipped 1 previous similar message [ 6293.604862] Lustre: server umount lustre-OST0000 complete [ 6299.627499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6299.635100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6299.654249] Lustre: Skipped 1 previous similar message [ 6299.680935] Lustre: Skipped 1 previous similar message [ 6304.737398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6304.747598] Lustre: Skipped 1 previous similar message [ 6309.856730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6309.865611] Lustre: Skipped 1 previous similar message [ 6311.391343] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6311.647129] Lustre: server umount lustre-MDT0000 complete [ 6314.990378] LustreError: 158577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6315.013990] LustreError: 158577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6315.086304] LustreError: 157008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121568 with bad export cookie 14229064300469720523 [ 6315.092863] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6315.096099] LustreError: 157008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6315.507050] Lustre: server umount lustre-MDT0001 complete [ 6319.321795] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 02:39:31 (1787121571) [ 6320.840702] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6322.462696] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 02:39:34 (1787121574) [ 6330.726346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6331.284759] LustreError: 161398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6331.442661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6336.489867] LustreError: 161399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6336.583484] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6341.603388] LustreError: 161398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6345.822817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6350.689590] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6358.615347] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6364.140759] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6370.926769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6371.253666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6371.258840] Lustre: Skipped 1 previous similar message [ 6377.722891] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6384.503028] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6401.010909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6401.014935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6401.028603] Lustre: Skipped 2 previous similar messages [ 6401.036789] Lustre: Skipped 1 previous similar message [ 6406.640392] Lustre: server umount lustre-MDT0000 complete [ 6409.187316] LustreError: 163890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6409.216287] LustreError: 163890:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6409.976941] LustreError: 161378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121663 with bad export cookie 14229064300469721405 [ 6409.984306] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6410.003043] LustreError: 161378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6414.304672] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6416.456782] Lustre: server umount lustre-MDT0001 complete [ 6422.677053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6433.552032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6433.738807] LustreError: 165262:0:(llog.c:1655:llog_backup()) MGC192.168.203.150@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6433.750495] Lustre: 165262:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.150@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6434.784727] LustreError: 165267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6434.801144] LustreError: 165267:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6451.515907] LustreError: 165267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6451.531103] LustreError: 165267:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6485.984434] LustreError: 165269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6486.012379] LustreError: 165269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [ 6495.199407] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6495.212339] Lustre: 165262:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.150@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6536.180756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6539.936557] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6541.290348] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6547.178760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6551.520800] LustreError: 165268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6551.532675] LustreError: 165268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 42 previous similar messages [ 6555.616124] LustreError: 148458:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bd5decc1 x1873932468748160/t0(0) o250->MGC192.168.203.150@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6559.940469] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6561.261226] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6566.184769] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6567.484516] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6572.875422] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6574.311460] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6591.973695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6591.976922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6591.980577] LustreError: Skipped 1 previous similar message [ 6592.006276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6592.013098] Lustre: Skipped 10 previous similar messages [ 6597.393960] Lustre: server umount lustre-MDT0000 complete [ 6600.506677] LustreError: 164203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121854 with bad export cookie 14229064300469727859 [ 6600.511373] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6600.522190] LustreError: 164203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6600.817295] Lustre: server umount lustre-MDT0001 complete [ 6614.545360] Lustre: server umount lustre-OST0000 complete [ 6634.074684] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 6644.900245] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 6674.728064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6674.900148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6674.919345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6674.977328] Lustre: lustre-MDT0000: new disk, initializing [ 6675.036736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6675.040993] Lustre: Skipped 1 previous similar message [ 6675.054287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6678.804992] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6688.145417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6688.310490] Lustre: 170364:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6688.593863] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6692.760157] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6702.095202] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6707.390889] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6713.219683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6713.466321] Lustre: lustre-OST0000: new disk, initializing [ 6713.472202] Lustre: Skipped 1 previous similar message [ 6713.478929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6713.485237] Lustre: Skipped 2 previous similar messages [ 6713.491429] Lustre: 172330:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6718.972473] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6719.554335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6719.564092] Lustre: Skipped 1 previous similar message [ 6719.573710] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6719.703116] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6727.316894] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6731.493507] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6731.654988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6735.794607] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6736.032501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6740.455570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6740.455724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6740.458102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6740.458109] Lustre: Skipped 3 previous similar messages [ 6740.491281] Lustre: Skipped 3 previous similar messages [ 6743.874289] Lustre: server umount lustre-OST0000 complete [ 6750.180483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6750.189170] LustreError: Skipped 1 previous similar message [ 6753.520555] Lustre: server umount lustre-MDT0000 complete [ 6755.809424] LustreError: 170371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6755.830471] LustreError: 170371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6756.881928] LustreError: 170355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122010 with bad export cookie 14229064300469744694 [ 6756.884977] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6756.893557] LustreError: 170355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6757.163325] Lustre: server umount lustre-MDT0001 complete [ 6763.449544] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 02:46:56 (1787122016) [ 6768.249032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6772.300306] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6779.085542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6782.766488] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6788.852293] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6793.649951] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6799.695562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6805.561066] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6811.918255] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6825.998662] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 02:47:58 (1787122078) [ 6831.083040] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6831.099957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6831.119240] Lustre: Skipped 2 previous similar messages [ 6831.128473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6831.133828] Lustre: Skipped 3 previous similar messages [ 6835.814108] Lustre: server umount lustre-OST0000 complete [ 6845.598518] Lustre: server umount lustre-MDT0000 complete [ 6848.718628] LustreError: 174720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122102 with bad export cookie 14229064300469745576 [ 6848.726494] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6848.729962] LustreError: 174720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6848.985577] Lustre: server umount lustre-MDT0001 complete [ 6859.147893] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 6861.533938] Key type lgssc unregistered [ 6861.840605] LNet: 178466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6861.850225] LNetError: 178466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6861.867168] LNet: Removed LNI 192.168.203.150@tcp [ 6862.814150] Key type .llcrypt unregistered [ 6862.816566] Key type ._llcrypt unregistered [ 6879.223510] Key type ._llcrypt registered [ 6879.226871] Key type .llcrypt registered [ 6879.317876] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 6880.173109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6880.258095] alg: No test for adler32 (adler32-zlib) [ 6881.250808] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 6881.443042] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 6883.082209] Key type lgssc registered [ 6884.077989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6890.719359] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6890.751477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6906.271490] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6911.391700] LustreError: 179718:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.150@tcp: failed processing log, type 4: rc = -110 [ 6916.511427] LustreError: 179718:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6916.523855] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6916.527899] LustreError: MGC192.168.203.150@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6916.534790] LustreError: 179718:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6916.541802] LustreError: 179718:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 6916.546696] LustreError: 179718:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 6916.602309] Lustre: server umount lustre-OST0000 complete [ 6916.607674] LustreError: 179718:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6924.201822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6925.556652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6929.298360] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6936.037896] LustreError: 180218:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6936.052145] LustreError: 180218:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6936.477238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6936.810311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6940.526171] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6946.472728] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6950.920514] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6956.578092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6956.816084] LustreError: 182032:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6956.826429] Lustre: 182032:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.150@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6956.960833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6961.949719] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6968.522293] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6970.215140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6970.222802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6970.229099] Lustre: Skipped 2 previous similar messages [ 6970.233212] Lustre: Skipped 1 previous similar message [ 6975.460275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6975.468709] Lustre: Skipped 3 previous similar messages [ 6976.498933] Lustre: server umount lustre-MDT0000 complete [ 6979.741840] LustreError: 180198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122233 with bad export cookie 18336057931415126930 [ 6979.751569] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6979.762121] LustreError: 180198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6980.197556] Lustre: server umount lustre-MDT0001 complete [ 6993.797152] Lustre: server umount lustre-OST0000 complete [ 7002.076077] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 7010.737717] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7042.726794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7043.001679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7043.032929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7043.109223] Lustre: lustre-MDT0000: new disk, initializing [ 7043.222483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7043.247047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7047.618130] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7056.964229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7057.037157] Lustre: 185556:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7057.067175] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7057.070721] Lustre: Skipped 1 previous similar message [ 7057.109206] Lustre: lustre-MDT0001: new disk, initializing [ 7057.165587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7057.186276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7057.190687] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7061.954319] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7071.464285] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7077.514612] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7083.933975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7084.168607] Lustre: lustre-OST0000: new disk, initializing [ 7084.171970] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7084.177885] Lustre: 187523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7084.257402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7090.889471] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7093.830728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7093.847610] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7094.027814] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7100.535173] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7105.205269] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7105.413558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7110.391309] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7110.645455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7114.727100] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7114.735810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7114.753199] LustreError: Skipped 1 previous similar message [ 7114.763410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7114.778763] Lustre: Skipped 1 previous similar message [ 7118.920741] Lustre: server umount lustre-OST0000 complete [ 7123.942177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7123.952604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7123.970190] Lustre: Skipped 1 previous similar message [ 7123.973319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7128.786493] Lustre: server umount lustre-MDT0000 complete [ 7130.086247] LustreError: 185564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7130.099499] LustreError: 185564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7132.210578] LustreError: 185548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122386 with bad export cookie 18336057931415127651 [ 7132.214987] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7132.218190] LustreError: 185548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7132.650909] Lustre: server umount lustre-MDT0001 complete [ 7139.670561] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 02:53:12 (1787122392) [ 7154.549074] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 7156.902216] Key type lgssc unregistered [ 7157.204732] LNet: 190468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7157.219297] LNetError: 190468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7157.248835] LNet: Removed LNI 192.168.203.150@tcp [ 7158.052821] Key type .llcrypt unregistered [ 7158.055370] Key type ._llcrypt unregistered [ 7172.110539] Key type ._llcrypt registered [ 7172.114077] Key type .llcrypt registered [ 7172.208624] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7172.933826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7173.009619] alg: No test for adler32 (adler32-zlib) [ 7174.066490] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 7174.324652] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 7175.983290] Key type lgssc registered [ 7176.771857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7183.113391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7183.134113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7184.527657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7188.492354] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7195.105283] LustreError: 191665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7195.119836] LustreError: 191665:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7195.857643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7196.215176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7200.569396] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7208.758935] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7215.875905] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7223.247632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7223.462453] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7223.464425] LustreError: 193530:0:(llog.c:1696:llog_backup()) MGC192.168.203.150@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7223.476089] Lustre: 193530:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.150@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7223.553166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7229.480779] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7236.371722] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7240.163137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7240.169964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7240.185580] Lustre: Skipped 2 previous similar messages [ 7240.199816] Lustre: Skipped 2 previous similar messages [ 7241.251783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7241.268924] Lustre: Skipped 2 previous similar messages [ 7246.310073] LustreError: 191665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7246.325030] LustreError: 191665:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7246.407386] Lustre: server umount lustre-MDT0000 complete [ 7249.494174] LustreError: 191644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122503 with bad export cookie 18058448670392973659 [ 7249.506597] LustreError: 192396:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000042f2a9f5 x1873933687091072/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7249.519861] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7249.852382] Lustre: server umount lustre-MDT0001 complete [ 7262.478454] Lustre: server umount lustre-OST0000 complete [ 7272.892953] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 7283.656989] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7319.504824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7319.786472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7319.824435] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7319.964381] Lustre: lustre-MDT0000: new disk, initializing [ 7320.039497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7320.066887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7324.933506] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7334.957375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7335.093260] Lustre: 197057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7335.173598] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7335.176858] Lustre: Skipped 1 previous similar message [ 7335.183895] LustreError: 197839:0:(llog.c:1696:llog_backup()) MGC192.168.203.150@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7335.194093] LustreError: 197839:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7335.200271] Lustre: 197839:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.150@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7335.220658] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7335.224940] Lustre: Skipped 1 previous similar message [ 7335.332250] Lustre: lustre-MDT0001: new disk, initializing [ 7335.450464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7335.491149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7335.505898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7339.708686] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7349.244204] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7356.082676] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7362.762476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7363.096041] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7363.100512] Lustre: Skipped 1 previous similar message [ 7363.103957] LustreError: 199000:0:(llog.c:1696:llog_backup()) MGC192.168.203.150@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7363.112972] LustreError: 199000:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7363.118436] Lustre: 199000:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.150@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7363.126450] Lustre: 199000:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7363.149987] Lustre: lustre-OST0000: new disk, initializing [ 7363.154938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7363.166128] Lustre: 199023:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7363.277210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7369.005824] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7370.276068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7370.284703] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7370.395162] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7377.189478] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7382.006600] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7382.216681] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7386.585458] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7386.788824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7390.700571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7390.708833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7390.716186] Lustre: Skipped 1 previous similar message [ 7390.737787] Lustre: Skipped 1 previous similar message [ 7394.699209] Lustre: server umount lustre-OST0000 complete [ 7400.929516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7400.954545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7400.968140] Lustre: Skipped 1 previous similar message [ 7404.706197] Lustre: server umount lustre-MDT0000 complete [ 7406.050142] LustreError: 197069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7406.085279] LustreError: 197069:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7409.418281] LustreError: 197050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122663 with bad export cookie 18058448670392974380 [ 7409.439813] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7409.448612] LustreError: 197050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7409.991394] Lustre: server umount lustre-MDT0001 complete [ 7420.309163] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 02:57:52 (1787122672) [ 7437.581918] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7447.469762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7447.772777] LustreError: 202152:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7447.835212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7452.433398] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7453.163313] LustreError: 202153:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7460.762735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7465.470136] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7468.264347] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7474.536615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7480.126683] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7487.745181] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7487.974080] Lustre: lustre-OST0001: new disk, initializing [ 7487.980838] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7488.005987] Lustre: 204459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7488.106901] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7488.112590] Lustre: Skipped 2 previous similar messages [ 7493.714966] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7497.813543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7497.833081] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7497.975755] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7505.321456] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7512.273844] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7514.552384] Lustre: DEBUG MARKER: ADD OST3 [ 7519.868136] loop: module loaded [ 7519.993105] loop0: detected capacity change from 0 to 204800000 [ 7520.279474] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7520.302989] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7520.325987] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7527.597349] loop0: detected capacity change from 0 to 204800000 [ 7528.091708] Lustre: DEBUG MARKER: STOP OST3 [ 7533.968442] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7534.229982] Lustre: lustre-OST0003: new disk, initializing [ 7534.234951] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7534.245878] Lustre: 206501:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7535.760242] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7536.060695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7536.082608] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7536.126202] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7537.442608] Lustre: Failing over lustre-OST0003 [ 7537.702310] Lustre: server umount lustre-OST0003 complete [ 7537.824814] LustreError: 206638:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7541.109659] LustreError: 204987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7541.124543] LustreError: 204987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7541.227812] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7541.235194] Lustre: Skipped 1 previous similar message [ 7541.241687] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7541.295949] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7550.585259] Lustre: DEBUG MARKER: START OST3 again [ 7557.959203] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7558.392876] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7558.400099] Lustre: Skipped 1 previous similar message [ 7558.432285] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7560.014789] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7560.353258] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7560.354242] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7565.655264] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7588.065448] Lustre: Failing over lustre-OST0003 [ 7588.176275] Lustre: server umount lustre-OST0003 complete [ 7591.394234] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7591.405314] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7591.418084] Lustre: Skipped 1 previous similar message [ 7591.426085] LustreError: 203636:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7591.448883] LustreError: 203636:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 7607.780691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7607.790918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7607.803297] Lustre: Skipped 4 previous similar messages [ 7607.806465] Lustre: Skipped 2 previous similar messages [ 7612.531360] Lustre: server umount lustre-MDT0000 complete [ 7615.830754] LustreError: 202132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122869 with bad export cookie 18058448670392975262 [ 7615.834362] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7615.838349] LustreError: 202132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7618.038028] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7618.043384] Lustre: Skipped 2 previous similar messages [ 7625.185286] LustreError: 202148:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7625.208047] LustreError: 202148:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 7630.303153] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7630.570482] Lustre: server umount lustre-MDT0001 complete [ 7633.862508] Lustre: server umount lustre-OST0000 complete [ 7637.334918] Lustre: server umount lustre-OST0001 complete [ 7643.589864] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 7651.191852] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7688.474579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7688.787357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7688.870907] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7689.047949] Lustre: lustre-MDT0000: new disk, initializing [ 7689.127283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7689.156388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7694.560691] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7705.847704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7706.027067] Lustre: 211931:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7706.082184] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7706.084678] Lustre: Skipped 1 previous similar message [ 7706.136261] Lustre: lustre-MDT0001: new disk, initializing [ 7706.251383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7706.265554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7710.916302] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7720.613771] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7726.479597] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7733.072508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7733.352655] Lustre: 213899:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7734.887868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7735.022938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7739.600560] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7749.421227] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7754.154877] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7754.431660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7758.746416] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7759.004827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7760.867387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7760.867863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7760.878751] Lustre: Skipped 2 previous similar messages [ 7760.879996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7760.880021] Lustre: Skipped 6 previous similar messages [ 7760.897256] LustreError: Skipped 2 previous similar messages [ 7775.199311] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7775.348759] Lustre: server umount lustre-OST0000 complete [ 7785.004299] Lustre: server umount lustre-MDT0000 complete [ 7786.464150] LustreError: 211939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7786.479231] LustreError: 211939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7787.839045] LustreError: 211925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123041 with bad export cookie 18058448670392977558 [ 7787.841356] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7787.857929] LustreError: 211925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7788.246330] Lustre: server umount lustre-MDT0001 complete [ 7791.120126] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 03:04:03 (1787123043) [ 7805.613381] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 7814.769782] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7845.731177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7846.112943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7846.180169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7846.186535] Lustre: Skipped 1 previous similar message [ 7846.292251] Lustre: lustre-MDT0000: new disk, initializing [ 7846.298710] Lustre: Skipped 1 previous similar message [ 7846.398632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7846.404394] Lustre: Skipped 1 previous similar message [ 7850.487541] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7860.365918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7860.499629] Lustre: 218510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7860.812714] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7864.838274] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7872.462590] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7877.669022] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7884.545225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7884.766051] Lustre: 220480:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7886.544328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7890.785917] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7900.162842] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7905.649981] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7905.839469] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7910.360871] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7910.573425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7917.028173] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7917.042992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7917.057310] Lustre: Skipped 3 previous similar messages [ 7917.066810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7917.073028] Lustre: Skipped 7 previous similar messages [ 7918.799083] Lustre: server umount lustre-OST0000 complete [ 7922.144616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7922.154055] LustreError: Skipped 1 previous similar message [ 7928.504094] Lustre: server umount lustre-MDT0000 complete [ 7931.817654] LustreError: 218504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123185 with bad export cookie 18058448670392978440 [ 7931.830149] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7931.834953] LustreError: 218504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7932.166949] Lustre: server umount lustre-MDT0001 complete [ 7940.331321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7940.819715] LustreError: 222698:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7946.306076] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7955.871800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7956.374435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7956.377350] Lustre: Skipped 6 previous similar messages [ 7960.180365] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7967.351000] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7972.836059] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7979.448942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7985.920092] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7993.208398] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8002.530508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8002.538593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8002.546532] Lustre: Skipped 3 previous similar messages [ 8002.559600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8002.563561] Lustre: Skipped 5 previous similar messages [ 8008.527814] Lustre: server umount lustre-OST0000 complete [ 8018.370786] Lustre: server umount lustre-MDT0000 complete [ 8022.562472] LustreError: 222680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123276 with bad export cookie 18058448670392979315 [ 8022.586098] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8023.095304] Lustre: server umount lustre-MDT0001 complete [ 8133.351572] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 8136.604886] Key type lgssc unregistered [ 8136.917152] LNet: 226360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8136.930446] LNetError: 226360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8136.959303] LNet: Removed LNI 192.168.203.150@tcp [ 8137.943163] Key type .llcrypt unregistered [ 8137.945561] Key type ._llcrypt unregistered [ 8164.784350] Key type ._llcrypt registered [ 8164.787968] Key type .llcrypt registered [ 8164.889353] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 8181.878950] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8182.936493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8182.956358] alg: No test for adler32 (adler32-zlib) [ 8184.156433] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 8184.363859] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 8186.007121] Key type lgssc registered [ 8187.174891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8224.076553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8224.136693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8225.488487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8225.528648] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8225.626895] Lustre: lustre-MDT0000: new disk, initializing [ 8225.754359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8225.772684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8230.383265] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8240.926508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8241.102922] Lustre: 229714:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8241.177931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8241.183121] Lustre: Skipped 1 previous similar message [ 8241.269841] Lustre: lustre-MDT0001: new disk, initializing [ 8241.377304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8241.524478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8241.534240] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8246.433955] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8255.770530] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8261.563714] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8268.088110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8268.400161] Lustre: lustre-OST0000: new disk, initializing [ 8268.405498] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8268.413280] Lustre: 231684:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8268.491808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8274.026222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8274.035871] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8274.195400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8274.521678] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8284.692118] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8290.097576] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8290.310899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8295.665757] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8296.002865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8300.010255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8300.010995] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8300.041235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8300.053356] Lustre: Skipped 1 previous similar message [ 8304.624751] Lustre: server umount lustre-OST0000 complete [ 8310.245220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8310.274705] Lustre: Skipped 1 previous similar message [ 8310.285191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8315.415670] Lustre: server umount lustre-MDT0000 complete [ 8320.366259] LustreError: 229707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123574 with bad export cookie 14619106462490793149 [ 8320.384884] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8320.399350] LustreError: 229707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8320.916756] Lustre: server umount lustre-MDT0001 complete [ 8342.537647] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8355.542554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8355.868755] LustreError: 234619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8355.882093] LustreError: 234619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8355.919667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8360.928767] LustreError: 234620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8361.839941] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8366.065759] LustreError: 234619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8371.173203] LustreError: 234620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8371.869499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8372.560832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8378.481944] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8381.351188] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8390.188860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8390.694910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8397.128149] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8405.948518] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8406.179680] Lustre: lustre-OST0001: new disk, initializing [ 8406.183278] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8406.188099] Lustre: 236928:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8412.818148] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8413.903300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8413.927508] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8414.023675] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8423.735778] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8435.580798] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8446.720176] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 03:14:59 (1787123699) [ 8449.504277] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8449.515744] LustreError: Skipped 1 previous similar message [ 8449.519465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8449.538483] Lustre: Skipped 1 previous similar message [ 8449.563479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8449.584126] Lustre: Skipped 1 previous similar message [ 8455.142605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8455.151614] Lustre: Skipped 4 previous similar messages [ 8460.256144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8460.262237] Lustre: Skipped 1 previous similar message [ 8463.329322] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8463.557853] Lustre: server umount lustre-MDT0000 complete [ 8465.382831] LustreError: 234615:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8465.402508] LustreError: 234615:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8467.195203] LustreError: 234600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123721 with bad export cookie 14619106462490794024 [ 8467.208792] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8467.215084] LustreError: 234600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8467.721976] Lustre: server umount lustre-MDT0001 complete [ 8481.967329] Lustre: server umount lustre-OST0000 complete [ 8496.085701] Lustre: server umount lustre-OST0001 complete [ 8503.011410] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 8514.091558] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8550.870141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8551.218336] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8551.240772] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8551.296876] Lustre: lustre-MDT0000: new disk, initializing [ 8551.359611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8551.364034] Lustre: Skipped 1 previous similar message [ 8551.375488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8555.240446] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8564.921267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8565.055371] Lustre: 240879:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8565.141695] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8565.153657] Lustre: Skipped 1 previous similar message [ 8565.270222] Lustre: lustre-MDT0001: new disk, initializing [ 8565.463757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8565.481467] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8570.203099] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8578.926660] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8583.865630] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8590.025977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8590.310087] Lustre: lustre-OST0000: new disk, initializing [ 8590.314383] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8590.324506] Lustre: 242849:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8590.439578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8590.447598] Lustre: Skipped 1 previous similar message [ 8591.994916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8592.002581] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8592.180669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8596.820979] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8605.519375] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8610.446565] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8610.637918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8615.739056] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8615.937920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8617.955125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8617.961952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8617.966250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8617.973941] Lustre: Skipped 4 previous similar messages [ 8617.985414] Lustre: Skipped 1 previous similar message [ 8632.287157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8632.464650] Lustre: server umount lustre-OST0000 complete [ 8637.407901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8637.413122] LustreError: Skipped 1 previous similar message [ 8637.421287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8637.437533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8637.446024] Lustre: Skipped 4 previous similar messages [ 8650.209142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8650.625339] Lustre: server umount lustre-MDT0000 complete [ 8653.803727] LustreError: 240890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8653.834670] LustreError: 240890:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8654.535501] LustreError: 240869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123908 with bad export cookie 14619106462490796173 [ 8654.540932] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8654.546303] LustreError: 240869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8654.894831] Lustre: server umount lustre-MDT0001 complete [ 8663.426467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8664.041741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8668.646813] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8673.252133] LustreError: 245068:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8673.300404] LustreError: 245068:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8678.131162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8678.370554] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8678.381379] Lustre: Skipped 6 previous similar messages [ 8683.799990] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8691.355946] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8697.150567] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8703.883829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8710.284452] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8717.133570] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8868.321218] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8868.329351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8868.333180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8868.340911] Lustre: Skipped 2 previous similar messages [ 8868.362800] Lustre: Skipped 1 previous similar message [ 8870.917201] Lustre: server umount lustre-OST0000 complete [ 8878.051970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8889.311157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8889.541541] Lustre: server umount lustre-MDT0000 complete [ 8893.352989] LustreError: 247688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124147 with bad export cookie 14619106462490797055 [ 8893.386076] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8893.799370] Lustre: server umount lustre-MDT0001 complete [ 8985.501905] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 8989.069203] Key type lgssc unregistered [ 8989.422651] LNet: 248674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8989.446726] LNetError: 248674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8989.485632] LNet: Removed LNI 192.168.203.150@tcp [ 8990.647144] Key type .llcrypt unregistered [ 8990.651848] Key type ._llcrypt unregistered [ 9016.381889] Key type ._llcrypt registered [ 9016.384917] Key type .llcrypt registered [ 9016.536258] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 9035.556914] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9036.768777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9036.806818] alg: No test for adler32 (adler32-zlib) [ 9037.764766] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 9038.065127] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 9039.751172] Key type lgssc registered [ 9040.730816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9081.906139] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9096.125393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9096.201328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9097.704964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9097.755176] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9097.898865] Lustre: lustre-MDT0000: new disk, initializing [ 9098.067099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9098.104147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9102.096509] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9112.847608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9112.937068] Lustre: 252749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9113.040774] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9113.044379] Lustre: Skipped 1 previous similar message [ 9113.100158] Lustre: lustre-MDT0001: new disk, initializing [ 9113.213540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9113.233237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9113.239216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9116.867782] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9122.137930] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9128.458381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9128.771818] Lustre: lustre-OST0000: new disk, initializing [ 9128.782068] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9128.791620] Lustre: 254378:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9128.935144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9130.888495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9130.900179] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9131.006500] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9134.302211] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9144.741124] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9144.925237] Lustre: lustre-OST0001: new disk, initializing [ 9144.933317] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9144.946805] Lustre: 255253:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9145.109414] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9151.767553] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9155.193280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9155.200804] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9155.365702] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9163.004479] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9170.536449] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9180.132229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9180.147868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9180.175550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9181.164896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9181.168671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9181.196311] Lustre: Skipped 2 previous similar messages [ 9185.955839] Lustre: server umount lustre-MDT0000 complete [ 9186.275376] LustreError: 252760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9186.309640] LustreError: 252760:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9188.888898] LustreError: 255252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124442 with bad export cookie 6765646419678328668 [ 9188.894410] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9188.904120] LustreError: 255252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9189.277218] Lustre: server umount lustre-MDT0001 complete [ 9203.821834] Lustre: server umount lustre-OST0000 complete [ 9207.777671] Lustre: 250256:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787124445/real 1787124445] req@ffff9c8bc1e6ca80 x1873935640634752/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787124461 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9207.817860] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9208.027603] Lustre: server umount lustre-OST0001 complete [ 9222.088568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9232.642862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9241.631891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9251.278193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9269.856820] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9280.333208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9280.363256] Lustre: lustre-MDT0000: reset Object Index mappings [ 9280.912241] LustreError: 260259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9281.012308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9285.219790] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9286.116474] LustreError: 260260:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9290.213896] LustreError: 260259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9293.296391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9293.318880] Lustre: lustre-MDT0001: reset Object Index mappings [ 9293.739971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9297.878541] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9300.673673] Lustre: 261400:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9307.268160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9314.450573] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9319.092295] LustreError: 261755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9323.582685] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9323.789487] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9323.794604] Lustre: Skipped 1 previous similar message [ 9325.883347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9325.945886] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9330.110365] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9337.453245] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9340.812361] Lustre: 263275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9345.019751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9345.039810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9345.055178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9345.057709] Lustre: Skipped 3 previous similar messages [ 9347.278243] Lustre: server umount lustre-MDT0000 complete [ 9350.485268] LustreError: 260241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124604 with bad export cookie 6765646419678335458 [ 9350.491722] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9350.499046] LustreError: 260241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9350.865809] Lustre: server umount lustre-MDT0001 complete [ 9364.704144] Lustre: server umount lustre-OST0000 complete [ 9367.650189] Lustre: 250253:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787124605/real 1787124605] req@ffff9c8cc78c3100 x1873935640681600/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787124621 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9367.682098] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9367.695038] Lustre: Skipped 3 previous similar messages [ 9368.289101] Lustre: server umount lustre-OST0001 complete [ 9385.778705] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:30:38 (1787124638) [ 9401.015680] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 9410.094802] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9443.158415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9443.489187] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9443.535807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9443.652876] Lustre: lustre-MDT0000: new disk, initializing [ 9443.778234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9443.796591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9449.414802] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9461.629138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9461.698651] Lustre: 267528:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9461.745506] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9461.750070] Lustre: Skipped 1 previous similar message [ 9461.794276] Lustre: lustre-MDT0001: new disk, initializing [ 9461.899082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9461.909431] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9466.360053] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9475.478922] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9481.125656] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9488.403029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9488.624975] Lustre: lustre-OST0000: new disk, initializing [ 9488.630619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9488.639574] Lustre: 269501:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9490.523206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9490.551638] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9490.645034] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9495.787835] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9506.373289] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9510.227653] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9510.457717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9514.820469] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9515.027351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9521.632956] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9521.634926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9521.637034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9521.637041] Lustre: Skipped 4 previous similar messages [ 9521.642145] LustreError: Skipped 1 previous similar message [ 9521.647693] Lustre: Skipped 1 previous similar message [ 9526.757247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9526.767127] Lustre: Skipped 2 previous similar messages [ 9531.359264] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9531.498206] Lustre: server umount lustre-OST0000 complete [ 9537.000513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9537.005722] Lustre: Skipped 1 previous similar message [ 9549.279409] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9549.477878] Lustre: server umount lustre-MDT0000 complete [ 9551.337991] LustreError: 268420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9551.357227] LustreError: 268420:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9553.221517] LustreError: 267520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124807 with bad export cookie 6765646419678337551 [ 9553.231275] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9553.239139] LustreError: 267520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9553.783049] Lustre: server umount lustre-MDT0001 complete [ 9572.659896] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9581.657392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9582.199293] LustreError: 272436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9582.251772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9582.257117] Lustre: Skipped 2 previous similar messages [ 9586.192445] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9594.780561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9600.200614] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9603.041164] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9609.029758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9614.416472] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9621.769558] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9621.951604] Lustre: lustre-OST0001: new disk, initializing [ 9621.959881] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9621.966622] Lustre: 274747:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9623.855870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9623.866712] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9623.943659] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9626.856911] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9635.168530] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9638.447723] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9643.101043] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9643.235975] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9651.681457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9651.693190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9651.710740] Lustre: Skipped 2 previous similar messages [ 9651.721649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9651.728762] Lustre: Skipped 4 previous similar messages [ 9654.995439] Lustre: server umount lustre-MDT0000 complete [ 9658.219968] LustreError: 274746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124912 with bad export cookie 6765646419678338433 [ 9658.228183] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9658.228510] LustreError: 274746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9658.524232] Lustre: server umount lustre-MDT0001 complete [ 9671.546510] Lustre: server umount lustre-OST0000 complete [ 9674.217432] Lustre: 250256:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787124912/real 1787124912] req@ffff9c8ce56ee300 x1873935640787584/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787124928 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9675.313731] Lustre: server umount lustre-OST0001 complete [ 9692.421875] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9701.729286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9702.096291] LustreError: 277804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9702.131724] LustreError: 277804:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9706.880744] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9716.019814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9716.342152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9716.346601] Lustre: Skipped 4 previous similar messages [ 9722.178590] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9724.947338] Lustre: 278944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9731.273734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9737.582229] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9746.118618] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9747.449886] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9747.496853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9752.702515] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9759.829952] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9762.892480] Lustre: 280820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9767.344662] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9767.471012] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9776.144485] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:37:08 (1787125028) [ 9792.993209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9792.998212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9793.009837] Lustre: Skipped 4 previous similar messages [ 9793.014045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9793.018351] Lustre: Skipped 4 previous similar messages [ 9796.974193] Lustre: server umount lustre-MDT0000 complete [ 9798.626401] LustreError: 277805:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9798.662340] LustreError: 277805:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9801.302469] LustreError: 277785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125055 with bad export cookie 6765646419678344971 [ 9801.307650] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9801.314559] LustreError: 277785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9807.892825] Lustre: server umount lustre-MDT0001 complete [ 9827.295106] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9827.396251] Lustre: server umount lustre-OST0000 complete [ 9837.080453] Lustre: server umount lustre-OST0001 complete [ 9843.677474] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 9853.502501] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9898.312419] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9907.259259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9907.447864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9907.481700] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9907.563810] Lustre: lustre-MDT0000: new disk, initializing [ 9907.695387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9911.781176] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9920.077172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9920.218239] Lustre: 285064:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9920.443199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9924.791549] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9936.119163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9936.492046] Lustre: 286697:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9938.126696] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9942.652265] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9950.897207] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9951.036519] Lustre: 287568:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9953.139299] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9953.144796] Lustre: Skipped 1 previous similar message [ 9953.261964] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9955.705826] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9964.244341] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9967.839239] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9967.847614] Lustre: Skipped 1 previous similar message [ 9980.887107] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 03:40:33 (1787125233) [ 9983.972157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9983.977551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9983.986142] Lustre: Skipped 8 previous similar messages [ 9983.998756] Lustre: Skipped 11 previous similar messages [ 9988.965620] Lustre: server umount lustre-MDT0000 complete [ 9989.091323] LustreError: 288452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9989.107769] LustreError: 288452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 9992.041773] LustreError: 285057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125245 with bad export cookie 6765646419678352622 [ 9992.048307] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9992.258844] Lustre: server umount lustre-MDT0001 complete [10006.166928] Lustre: server umount lustre-OST0000 complete [10011.103179] Lustre: 250255:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787125248/real 1787125248] req@ffff9c8bcb753100 x1873935640929152/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787125264 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10011.642608] Lustre: server umount lustre-OST0001 complete [10022.601331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10023.295241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10023.298650] Lustre: Skipped 6 previous similar messages [10028.760553] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10037.770901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10042.895129] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10049.932996] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10055.272431] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10061.827520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10067.992786] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10075.098486] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10090.795662] loop1: detected capacity change from 0 to 204800000 [10091.062119] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10091.081131] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10091.102129] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10096.125498] loop1: detected capacity change from 0 to 204800000 [10100.983437] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10101.062315] LustreError: 289800:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs19083: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10101.077199] LustreError: 289800:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs19083 [10101.084141] LustreError: 289800:0:(mgs_handler.c:517:mgs_target_reg()) fs19083-MDT0001: Error checking target index: rc = -13 [10101.099568] LustreError: fs19083-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10101.111746] LustreError: 293530:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10101.118241] LustreError: 293530:0:(tgt_mount.c:2129:server_put_super()) no obd fs19083-MDT0001 [10101.122449] LustreError: 293530:0:(tgt_mount.c:116:server_deregister_mount()) fs19083-MDT0001 not registered [10101.163452] Lustre: server umount fs19083-MDT0001 complete [10101.166363] LustreError: 293530:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10109.337579] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10109.491331] Lustre: Setting parameter fs19083-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs19083-MDT0001 [10109.544462] Lustre: srv-fs19083-MDT0001: No data found on store. Initialize space. [10109.551043] Lustre: Skipped 4 previous similar messages [10109.648319] Lustre: fs19083-MDT0001: new disk, initializing [10109.652679] Lustre: Skipped 3 previous similar messages [10109.805910] Lustre: srv-fs19083-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10109.818844] Lustre: 293998:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs19083-MDT0001-mdtlov: can't create new llog: rc = -115 [10109.832982] LustreError: 293998:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs19083-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10113.825942] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10119.856490] Lustre: server umount fs19083-MDT0001 complete [10132.315099] loop1: detected capacity change from 0 to 204800000 [10138.026790] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10138.560697] Lustre: 295565:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs19083-MDT0001-mdtlov: can't create new llog: rc = -115 [10138.573207] LustreError: 295565:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs19083-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10143.257404] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10148.504448] Lustre: server umount fs19083-MDT0001 complete [10150.396698] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10160.614232] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10160.620582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10160.636605] Lustre: Skipped 3 previous similar messages [10160.656012] Lustre: Skipped 2 previous similar messages [10172.383132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10172.563960] Lustre: server umount lustre-MDT0000 complete [10180.258570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10180.375377] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10185.577361] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10185.716176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10193.070973] loop1: detected capacity change from 0 to 204800000 [10198.238440] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10198.745461] Lustre: 298012:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs19083-MDT0001-mdtlov: can't create new llog: rc = -115 [10198.761231] LustreError: 298012:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs19083-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10203.362305] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10208.084453] Lustre: server umount fs19083-MDT0001 complete [10231.787838] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10231.819743] LustreError: Skipped 1 previous similar message [10236.653541] Lustre: server umount lustre-OST0000 complete [10254.305810] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10254.436782] Lustre: server umount lustre-MDT0000 complete [10256.361765] LustreError: 289809:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10256.380905] LustreError: 289809:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 102 previous similar messages [10257.668866] LustreError: 289794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125511 with bad export cookie 6765646419678355303 [10257.669898] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10257.687629] LustreError: 289794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10258.007699] Lustre: server umount lustre-MDT0001 complete [10268.319976] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [10271.227700] Key type lgssc unregistered [10271.528607] LNet: 300359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10271.539531] LNetError: 300359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10271.572282] LNet: Removed LNI 192.168.203.150@tcp [10272.381318] Key type .llcrypt unregistered [10272.384798] Key type ._llcrypt unregistered [10289.059880] Key type ._llcrypt registered [10289.062459] Key type .llcrypt registered [10289.159634] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [10290.242118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10290.261106] alg: No test for adler32 (adler32-zlib) [10291.358891] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [10291.665026] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [10293.343152] Key type lgssc registered [10294.783693] Lustre: Echo OBD driver; http://www.lustre.org/ [10302.626492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10302.655239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10304.281768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10308.916836] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10314.725427] LustreError: 301612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10314.759242] LustreError: 301612:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10316.212355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10316.735508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10320.556235] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10327.042910] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10332.564152] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10338.629323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10339.097684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10341.181508] LustreError: 303442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10345.794290] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10346.474771] LustreError: 303441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10349.544353] LustreError: 303442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10349.565780] LustreError: 303442:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10353.259442] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10354.674825] LustreError: 303577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10354.703488] LustreError: 303577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10364.903628] LustreError: 303442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10364.921772] LustreError: 303442:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10368.644274] loop1: detected capacity change from 0 to 204800000 [10368.777737] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10368.793177] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10368.804473] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10373.101441] loop1: detected capacity change from 0 to 204800000 [10378.170552] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10378.374884] Lustre: fs859-OST0000: new disk, initializing [10378.378861] Lustre: srv-fs859-OST0000: No data found on store. Initialize space. [10378.387813] Lustre: 305346:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10378.453486] Lustre: fs859-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10381.648764] LustreError: 303442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10381.665416] LustreError: 303442:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [10384.194463] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10392.223363] Lustre: server umount fs859-OST0000 complete [10403.353448] loop1: detected capacity change from 0 to 204800000 [10408.302524] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10408.541497] Lustre: fs859-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10414.284637] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10416.108923] LustreError: 303441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10416.118248] LustreError: 303441:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [10422.004102] loop2: detected capacity change from 0 to 204800000 [10422.183932] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10422.199271] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10422.211893] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10426.353029] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10427.205086] loop2: detected capacity change from 0 to 204800000 [10431.461110] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10435.003509] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10435.199069] LustreError: fs859-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10435.217359] LustreError: fs859-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10435.238291] LustreError: 308387:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10435.243456] LustreError: 308387:0:(tgt_mount.c:2129:server_put_super()) no obd fs859-OST0001 [10435.250610] LustreError: 308387:0:(tgt_mount.c:116:server_deregister_mount()) fs859-OST0001 not registered [10435.326055] Lustre: server umount fs859-OST0001 complete [10435.327812] LustreError: 308387:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10448.648702] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10448.876507] Lustre: fs859-OST0001: new disk, initializing [10448.884434] Lustre: srv-fs859-OST0001: No data found on store. Initialize space. [10448.901636] Lustre: 308939:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10448.981696] Lustre: fs859-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10455.124378] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10462.950645] Lustre: server umount fs859-OST0000 complete [10470.850838] Lustre: server umount fs859-OST0001 complete [10482.048816] loop1: detected capacity change from 0 to 204800000 [10482.661825] LustreError: 303441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10482.686846] LustreError: 303441:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [10487.216448] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10487.464806] Lustre: fs859-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10493.841281] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10501.387996] loop2: detected capacity change from 0 to 204800000 [10507.433857] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10514.083995] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10523.436773] Lustre: server umount fs859-OST0000 complete [10533.225292] Lustre: server umount fs859-OST0001 complete [10546.144176] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10546.159690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10546.172860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10549.219870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10549.223472] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10549.224564] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10551.115454] Lustre: server umount lustre-OST0000 complete [10554.336892] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10554.346583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10554.350785] Lustre: Skipped 1 previous similar message [10559.457377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10559.478189] Lustre: Skipped 1 previous similar message [10560.664861] Lustre: server umount lustre-MDT0000 complete [10564.489403] LustreError: 301592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125818 with bad export cookie 18388374897525085141 [10564.496080] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10564.503017] LustreError: 301592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10564.933731] Lustre: server umount lustre-MDT0001 complete [10575.886600] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [10578.839690] Key type lgssc unregistered [10579.130238] LNet: 314862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10579.148541] LNetError: 314862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10579.161827] LNet: Removed LNI 192.168.203.150@tcp [10580.042155] Key type .llcrypt unregistered [10580.045988] Key type ._llcrypt unregistered [10611.974773] Key type ._llcrypt registered [10611.982962] Key type .llcrypt registered [10612.115872] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [10627.673570] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [10628.286637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10628.342511] alg: No test for adler32 (adler32-zlib) [10629.318995] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [10629.551887] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [10631.223238] Key type lgssc registered [10632.023872] Lustre: Echo OBD driver; http://www.lustre.org/ [10670.506765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10670.589303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10672.060042] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10672.130917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10672.281834] Lustre: lustre-MDT0000: new disk, initializing [10672.431796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10672.462295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10677.184976] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10687.017378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10687.220627] Lustre: 318722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10687.323267] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10687.338390] Lustre: Skipped 1 previous similar message [10687.471432] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10687.476124] Lustre: lustre-MDT0001: new disk, initializing [10687.662852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10687.703317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10687.734926] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10692.539530] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10701.595753] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10707.179067] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10713.447457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10713.726602] Lustre: lustre-OST0000: new disk, initializing [10713.735475] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10713.750978] Lustre: 320688:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10713.852685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10719.290496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10719.298624] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10719.417552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10720.021236] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10728.836569] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10730.692169] Lustre: Mounted lustre-client [10731.777090] Lustre: Unmounted lustre-client [10741.228220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10741.245154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10741.249126] Lustre: Skipped 1 previous similar message [10746.342736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10746.347526] Lustre: Skipped 1 previous similar message [10746.538592] Lustre: server umount lustre-OST0000 complete [10751.460371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10751.462855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10751.482518] Lustre: Skipped 2 previous similar messages [10756.106326] Lustre: server umount lustre-MDT0000 complete [10759.420219] LustreError: 319516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126013 with bad export cookie 8889045887584834645 [10759.421572] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10759.425584] LustreError: 319516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10759.595914] Lustre: server umount lustre-MDT0001 complete [10769.812954] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [10773.581386] Key type lgssc unregistered [10785.247815] LNet: 322848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10785.257558] LNetError: 322848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10860.075651] LNet: Removed LNI 192.168.203.150@tcp [10861.038508] Key type .llcrypt unregistered [10861.040900] Key type ._llcrypt unregistered [10878.886384] Key type ._llcrypt registered [10878.890994] Key type .llcrypt registered [10878.990972] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [10894.471332] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [10895.554133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10895.688063] alg: No test for adler32 (adler32-zlib) [10897.006680] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [10897.418572] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [10899.215162] Key type lgssc registered [10900.210340] Lustre: Echo OBD driver; http://www.lustre.org/ [10930.708655] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 03:56:23 (1787126183) [10937.018677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10937.060364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10938.404849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10938.479814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10938.610926] Lustre: lustre-MDT0000: new disk, initializing [10938.729289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10938.766135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10943.331721] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10954.050270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10954.202527] Lustre: 326069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10954.257453] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10954.262807] Lustre: Skipped 1 previous similar message [10954.351728] Lustre: lustre-MDT0001: new disk, initializing [10954.475757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10954.537233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10954.556915] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10958.917363] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10968.445689] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10974.097880] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10981.174156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10981.485629] Lustre: lustre-OST0000: new disk, initializing [10981.490858] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10981.510630] Lustre: 328041:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10981.621661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10987.626334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10987.644050] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10987.749060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10988.279755] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10997.203495] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11013.614203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11013.635972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11013.636475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11013.664292] Lustre: Skipped 1 previous similar message [11018.607049] Lustre: server umount lustre-OST0000 complete [11026.400431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11026.415092] LustreError: Skipped 1 previous similar message [11026.421239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11026.440452] Lustre: Skipped 1 previous similar message [11026.446172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11028.960818] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11028.963618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11029.004115] Lustre: Skipped 1 previous similar message [11030.719285] Lustre: server umount lustre-MDT0000 complete [11036.023243] LustreError: 328040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126289 with bad export cookie 7183830724331012816 [11036.027290] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11036.032723] LustreError: 328040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11036.310328] Lustre: server umount lustre-MDT0001 complete [11046.593977] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [11048.981704] Key type lgssc unregistered [11049.217512] LNet: 329950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11049.224569] LNetError: 329950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11049.238063] LNet: Removed LNI 192.168.203.150@tcp [11050.230608] Key type .llcrypt unregistered [11050.235072] Key type ._llcrypt unregistered [11071.531222] Key type ._llcrypt registered [11071.533402] Key type .llcrypt registered [11071.613957] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [11073.303644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11073.397029] alg: No test for adler32 (adler32-zlib) [11074.478357] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [11074.757342] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [11076.535836] Key type lgssc registered [11077.874379] Lustre: Echo OBD driver; http://www.lustre.org/ [11085.886021] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11085.909510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11087.428282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11092.088187] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11097.569325] LustreError: 331226:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11097.615216] LustreError: 331226:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11101.612469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11101.989389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11107.753246] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11115.801243] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11122.614420] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11129.247708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11129.681468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11136.264230] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11143.120266] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11156.602761] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11158.072748] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11162.596894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11162.611255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11162.612555] Lustre: Skipped 2 previous similar messages [11162.626703] Lustre: Skipped 2 previous similar messages [11166.548178] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.50@tcp (stopping) [11167.715956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11167.721830] Lustre: Skipped 2 previous similar messages [11167.820828] Lustre: server umount lustre-MDT0000 complete [11171.199323] LustreError: 331206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126425 with bad export cookie 14261580329257769654 [11171.216570] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11171.220232] LustreError: 331206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11171.647564] LustreError: 332370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11171.681482] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.50@tcp (stopping) [11172.836802] LustreError: 331219:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11172.838391] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11176.763550] LustreError: 332370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11176.779115] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.50@tcp (stopping) [11176.788039] Lustre: Skipped 1 previous similar message [11177.618338] Lustre: server umount lustre-MDT0001 complete [11180.948668] Lustre: server umount lustre-OST0000 complete [11189.507879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11194.038846] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11196.747866] Lustre: 335406:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.150@tcp [11198.666824] Lustre: server umount MGS complete [11206.579169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11207.214742] LustreError: 335978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11207.234816] LustreError: 335978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11207.299713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11212.562589] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11217.890783] LustreError: 335979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11217.904044] LustreError: 335979:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11221.045315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11221.539912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11226.093354] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11233.614495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11233.884628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11239.398935] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11245.881770] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11247.196553] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11252.463369] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11258.234888] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [11262.432204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11262.433324] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11262.441860] LustreError: Skipped 1 previous similar message [11262.445277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11262.466643] Lustre: Skipped 1 previous similar message [11262.494836] Lustre: Skipped 2 previous similar messages [11264.546713] Lustre: server umount lustre-OST0000 complete [11272.672064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11272.677990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11272.712179] Lustre: Skipped 1 previous similar message [11282.399318] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11282.527256] Lustre: server umount lustre-MDT0000 complete [11282.919444] LustreError: 336718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11282.946799] LustreError: 336718:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11285.612105] LustreError: 335960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126539 with bad export cookie 14261580329257770494 [11285.622063] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11285.629142] LustreError: 335960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11285.900118] Lustre: server umount lustre-MDT0001 complete [11296.294177] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [11299.376208] Key type lgssc unregistered [11299.724852] LNet: 339424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11299.737357] LNetError: 339424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11300.773888] LNet: Removed LNI 192.168.203.150@tcp [11301.756570] Key type .llcrypt unregistered [11301.765285] Key type ._llcrypt unregistered [11319.486154] Key type ._llcrypt registered [11319.488956] Key type .llcrypt registered [11319.668633] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [11335.027188] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [11336.258272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11336.292175] alg: No test for adler32 (adler32-zlib) [11337.307333] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [11337.540422] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [11339.231171] Key type lgssc registered [11340.262714] Lustre: Echo OBD driver; http://www.lustre.org/ [11375.830245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11375.846589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11377.216812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11377.271483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11377.357459] Lustre: lustre-MDT0000: new disk, initializing [11377.427116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11377.459990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11381.918877] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11392.534220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11392.619225] Lustre: 342587:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11392.666407] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11392.669757] Lustre: Skipped 1 previous similar message [11392.737246] Lustre: lustre-MDT0001: new disk, initializing [11392.816965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11392.863527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11392.880413] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11397.660308] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11407.283376] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11413.259150] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11420.296313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11420.602232] Lustre: lustre-OST0000: new disk, initializing [11420.607168] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11420.618938] Lustre: 344555:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11420.694823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11426.914522] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11428.937258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11428.953389] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11429.041446] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11436.800984] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11441.789735] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11441.997399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11446.550984] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11446.796179] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11449.824644] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11449.831331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11449.837468] LustreError: Skipped 1 previous similar message [11449.837895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11449.885901] Lustre: Skipped 1 previous similar message [11454.951569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11454.961409] Lustre: Skipped 2 previous similar messages [11460.074046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11460.082762] Lustre: Skipped 1 previous similar message [11463.136052] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11463.308100] Lustre: server umount lustre-OST0000 complete [11469.804524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11469.814308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11469.830051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11473.013193] Lustre: server umount lustre-MDT0000 complete [11476.452861] LustreError: 345893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126730 with bad export cookie 8869618118832807902 [11476.458214] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11476.463513] LustreError: 345893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11476.872500] Lustre: server umount lustre-MDT0001 complete [11485.493653] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 04:05:37 (1787126737) [11501.480579] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [11512.868830] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [11550.510735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11550.849303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11550.907315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11551.024732] Lustre: lustre-MDT0000: new disk, initializing [11551.089487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11551.108603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11556.881994] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11568.646435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11568.825689] Lustre: 349314:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11568.906402] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11568.912205] Lustre: Skipped 1 previous similar message [11569.031438] Lustre: lustre-MDT0001: new disk, initializing [11569.129782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11569.188715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11569.207075] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11574.061291] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11583.270141] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11589.445347] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11596.010495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11596.297197] Lustre: lustre-OST0000: new disk, initializing [11596.299830] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11596.310803] Lustre: 351282:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11596.392440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11602.429438] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11603.034638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11603.050528] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11603.195964] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11611.921199] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11649.508214] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11649.509092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11649.512542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11649.512552] Lustre: Skipped 2 previous similar messages [11649.522932] LustreError: Skipped 1 previous similar message [11649.541085] Lustre: Skipped 2 previous similar messages [11659.754030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11659.760440] Lustre: Skipped 4 previous similar messages [11660.255144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11660.372964] Lustre: server umount lustre-OST0000 complete [11664.865781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11664.876457] Lustre: Skipped 1 previous similar message [11670.369475] Lustre: server umount lustre-MDT0000 complete [11674.472631] LustreError: 351281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126928 with bad export cookie 8869618118832808784 [11674.475481] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11674.481673] LustreError: 351281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11674.826778] Lustre: server umount lustre-MDT0001 complete [11688.542344] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [11690.857821] Key type lgssc unregistered [11691.180647] LNet: 353462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11691.198407] LNetError: 353462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11692.264778] LNet: Removed LNI 192.168.203.150@tcp [11693.037481] Key type .llcrypt unregistered [11693.047213] Key type ._llcrypt unregistered [11780.794478] Key type ._llcrypt registered [11780.797695] Key type .llcrypt registered [11780.878761] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [11781.130397] Key type .llcrypt unregistered [11781.135041] Key type ._llcrypt unregistered [11859.184441] Key type ._llcrypt registered [11859.187251] Key type .llcrypt registered [11859.286283] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [11859.616295] Key type .llcrypt unregistered [11859.623109] Key type ._llcrypt unregistered [11886.438803] Key type ._llcrypt registered [11886.441433] Key type .llcrypt registered [11886.560690] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [11886.926294] Key type .llcrypt unregistered [11886.934738] Key type ._llcrypt unregistered [11915.230171] Key type ._llcrypt registered [11915.232606] Key type .llcrypt registered [11915.327240] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [11915.706312] Key type .llcrypt unregistered [11915.718963] Key type ._llcrypt unregistered [11993.428748] Key type ._llcrypt registered [11993.434256] Key type .llcrypt registered [11993.562460] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [11993.812616] Key type .llcrypt unregistered [11993.816886] Key type ._llcrypt unregistered [12022.610808] Key type ._llcrypt registered [12022.612940] Key type .llcrypt registered [12022.701598] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [12022.962622] Key type .llcrypt unregistered [12022.965122] Key type ._llcrypt unregistered [12053.607933] Key type ._llcrypt registered [12053.613812] Key type .llcrypt registered [12053.755663] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [12054.025295] Key type .llcrypt unregistered [12054.027358] Key type ._llcrypt unregistered [12081.028647] Key type ._llcrypt registered [12081.031529] Key type .llcrypt registered [12081.172955] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [12081.490648] Key type .llcrypt unregistered [12081.492404] Key type ._llcrypt unregistered [12108.650325] Key type ._llcrypt registered [12108.653318] Key type .llcrypt registered [12108.790520] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [12109.056276] Key type .llcrypt unregistered [12109.059482] Key type ._llcrypt unregistered [12136.954245] Key type ._llcrypt registered [12136.956077] Key type .llcrypt registered [12137.056176] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [12137.342270] Key type .llcrypt unregistered [12137.344576] Key type ._llcrypt unregistered [12202.072329] Key type ._llcrypt registered [12202.083704] Key type .llcrypt registered [12202.229442] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [12202.547606] Key type .llcrypt unregistered [12202.550307] Key type ._llcrypt unregistered [12234.371421] Key type ._llcrypt registered [12234.375593] Key type .llcrypt registered [12234.516379] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [12234.827591] Key type .llcrypt unregistered [12234.829466] Key type ._llcrypt unregistered [12267.250656] Key type ._llcrypt registered [12267.252877] Key type .llcrypt registered [12267.386346] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [12267.739529] Key type .llcrypt unregistered [12267.743571] Key type ._llcrypt unregistered [12320.126218] Key type ._llcrypt registered [12320.129624] Key type .llcrypt registered [12320.253821] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [12336.758923] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [12337.735034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12337.787861] alg: No test for adler32 (adler32-zlib) [12339.112819] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [12339.429446] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [12341.072433] Key type lgssc registered [12342.482841] Lustre: Echo OBD driver; http://www.lustre.org/ [12379.094189] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12379.146697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12380.474483] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12380.536458] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12380.645874] Lustre: tst250fs-MDT0000: new disk, initializing [12380.742056] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12380.771542] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12386.141535] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12395.893126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12396.052765] Lustre: 373745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12396.147899] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12396.165105] Lustre: Skipped 1 previous similar message [12396.280554] Lustre: tst250fs-MDT0001: new disk, initializing [12396.405508] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12396.449638] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12396.464375] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12401.333585] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12406.429949] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12413.044917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12413.274949] Lustre: tst250fs-OST0000: new disk, initializing [12413.278343] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12413.285062] Lustre: 375377:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12413.392626] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12419.439974] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12420.680629] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12420.691911] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12420.839912] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12429.155932] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12429.343703] Lustre: tst250fs-OST0001: new disk, initializing [12429.351562] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12429.365697] Lustre: 376248:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12429.481777] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12435.065460] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12438.655694] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12438.667288] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12438.784673] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12440.375610] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12443.088423] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12443.095836] Lustre: Skipped 2 previous similar messages [12463.075242] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12463.091688] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12463.111803] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12464.616952] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12464.622403] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12464.634913] Lustre: Skipped 1 previous similar message [12464.648610] Lustre: Skipped 3 previous similar messages [12469.749539] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12469.764241] Lustre: Skipped 2 previous similar messages [12474.868491] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12474.890055] Lustre: Skipped 3 previous similar messages [12475.359192] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12475.940702] Lustre: server umount tst250fs-MDT0000 complete [12479.768684] LustreError: 373738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127733 with bad export cookie 9163390271764143736 [12479.781161] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12479.981767] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12479.994691] Lustre: Skipped 1 previous similar message [12480.246624] Lustre: server umount tst250fs-MDT0001 complete [12486.469343] Lustre: server umount tst250fs-OST0000 complete [12490.650440] Lustre: server umount tst250fs-OST0001 complete [12510.639939] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [12520.038102] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [12552.770559] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 04:23:25 (1787127805) [12554.888934] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12574.400300] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [12587.258762] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [12619.727222] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12364 sec ========= 04:24:31 (1787127871) [12621.463819] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:24:33 (1787127873) === [12624.798340] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:24:37 (1787127877) === [12661.591828] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [12664.567271] Key type lgssc unregistered [12664.842676] LNet: 385376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12664.849410] LNetError: 385376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12665.893695] LNet: Removed LNI 192.168.203.150@tcp [12666.904312] Key type .llcrypt unregistered [12666.911791] Key type ._llcrypt unregistered