[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 502941687 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003144] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010136] LSM: Security Framework initializing [ 0.011057] Yama: becoming mindful. [ 0.012046] SELinux: Initializing. [ 0.013068] *** VALIDATE selinux *** [ 0.021750] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026756] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027147] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029093] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030110] *** VALIDATE tmpfs *** [ 0.031462] *** VALIDATE proc *** [ 0.033081] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.035280] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038023] Spectre V2 : User space: Vulnerable [ 0.039005] Speculative Store Bypass: Vulnerable [ 0.041874] debug: unmapping init [mem 0xffffffffb8459000-0xffffffffb8460fff] [ 0.043841] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044552] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045018] ... version: 2 [ 0.045789] ... bit width: 48 [ 0.046008] ... generic registers: 4 [ 0.046813] ... value mask: 0000ffffffffffff [ 0.047011] ... max period: 00007fffffffffff [ 0.048010] ... fixed-purpose events: 3 [ 0.048931] ... event mask: 000000070000000f [ 0.049292] rcu: Hierarchical SRCU implementation. [ 0.051762] smp: Bringing up secondary CPUs ... [ 0.052541] x86: Booting SMP configuration: [ 0.053016] .... node #0, CPUs: #1 #2 #3 [ 0.056435] smp: Brought up 1 node, 4 CPUs [ 0.057742] smpboot: Max logical packages: 1 [ 0.058009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138920] node 0 deferred pages initialised in 77ms [ 0.142190] devtmpfs: initialized [ 0.143221] x86/mm: Memory block size: 128MB [ 0.145795] gcov: version magic: 0x41383552 [ 0.147364] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154369] pinctrl core: initialized pinctrl subsystem [ 0.156187] [ 0.156806] ************************************************************* [ 0.159012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161012] ** ** [ 0.164013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166013] ** ** [ 0.168011] ** This means that this kernel is built to expose internal ** [ 0.171138] ** IOMMU data structures, which may compromise security on ** [ 0.173012] ** your system. ** [ 0.177021] ** ** [ 0.182016] ** If you see this message and you are not debugging the ** [ 0.184010] ** kernel, report this immediately to your vendor! ** [ 0.186011] ** ** [ 0.188011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.190011] ************************************************************* [ 0.193051] NET: Registered protocol family 16 [ 0.195783] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.199066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.203072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.209038] cpuidle: using governor menu [ 0.211000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.213431] PCI: Using configuration type 1 for base access [ 0.215137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.220048] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.223024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.227080] cryptd: max_cpu_qlen set to 1000 [ 0.230243] ACPI: Added _OSI(Module Device) [ 0.232017] ACPI: Added _OSI(Processor Device) [ 0.234012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.236069] ACPI: Added _OSI(Processor Aggregator Device) [ 0.242643] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.249760] ACPI: Interpreter enabled [ 0.251097] ACPI: PM: (supports S0 S3 S4 S5) [ 0.253018] ACPI: Using IOAPIC for interrupt routing [ 0.255123] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.258464] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.272778] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.275046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.278020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.282085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.287843] acpiphp: Slot [2] registered [ 0.290113] acpiphp: Slot [5] registered [ 0.292233] acpiphp: Slot [6] registered [ 0.293129] acpiphp: Slot [7] registered [ 0.295118] acpiphp: Slot [8] registered [ 0.296082] acpiphp: Slot [9] registered [ 0.298212] acpiphp: Slot [10] registered [ 0.300170] acpiphp: Slot [3] registered [ 0.302149] acpiphp: Slot [4] registered [ 0.304111] acpiphp: Slot [11] registered [ 0.305126] acpiphp: Slot [12] registered [ 0.307136] acpiphp: Slot [13] registered [ 0.308111] acpiphp: Slot [14] registered [ 0.310140] acpiphp: Slot [15] registered [ 0.312097] acpiphp: Slot [16] registered [ 0.314121] acpiphp: Slot [17] registered [ 0.315132] acpiphp: Slot [18] registered [ 0.317112] acpiphp: Slot [19] registered [ 0.318164] acpiphp: Slot [20] registered [ 0.320164] acpiphp: Slot [21] registered [ 0.322099] acpiphp: Slot [22] registered [ 0.323090] acpiphp: Slot [23] registered [ 0.325212] acpiphp: Slot [24] registered [ 0.327106] acpiphp: Slot [25] registered [ 0.329124] acpiphp: Slot [26] registered [ 0.330116] acpiphp: Slot [27] registered [ 0.332106] acpiphp: Slot [28] registered [ 0.333121] acpiphp: Slot [29] registered [ 0.335130] acpiphp: Slot [30] registered [ 0.337115] acpiphp: Slot [31] registered [ 0.339069] PCI host bridge to bus 0000:00 [ 0.340017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.342022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.345026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.347036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.350027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.353032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.355195] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.358059] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.361235] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.374007] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.381062] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.383018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.385020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.388047] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.392462] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.394928] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.398042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.401744] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.406026] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.420015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.425013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.431943] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.445020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.459021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.494018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.503441] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.517017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.529017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.548054] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.558565] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.563012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.569015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.584022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.594000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.609018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.623018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.660018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.684562] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.698017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.712023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.746016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.761845] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.770016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.781029] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.824020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.848822] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.851416] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.854315] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.856441] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.859323] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.865097] iommu: Default domain type: Passthrough [ 0.868345] SCSI subsystem initialized [ 0.869156] ACPI: bus type USB registered [ 0.871108] usbcore: registered new interface driver usbfs [ 0.874118] usbcore: registered new interface driver hub [ 0.876092] usbcore: registered new device driver usb [ 0.878277] pps_core: LinuxPPS API ver. 1 registered [ 0.880023] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.884093] PTP clock support registered [ 0.886125] EDAC MC: Ver: 3.0.0 [ 0.888139] PCI: Using ACPI for IRQ routing [ 0.891028] NetLabel: Initializing [ 0.892010] NetLabel: domain hash size = 128 [ 0.895013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.898073] NetLabel: unlabeled traffic allowed by default [ 0.900137] vgaarb: loaded [ 0.901250] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.902016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.908661] clocksource: Switched to clocksource kvm-clock [ 1.014875] VFS: Disk quotas dquot_6.6.0 [ 1.016456] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.018429] *** VALIDATE ramfs *** [ 1.019341] *** VALIDATE hugetlbfs *** [ 1.020677] pnp: PnP ACPI init [ 1.022699] pnp: PnP ACPI: found 6 devices [ 1.039834] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.041960] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.043403] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.044700] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.046236] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.047843] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.050442] NET: Registered protocol family 2 [ 1.052855] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.056964] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.059278] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.065258] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.068927] TCP: Hash tables configured (established 65536 bind 65536) [ 1.072147] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.075505] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.078661] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.081810] NET: Registered protocol family 1 [ 1.084704] RPC: Registered named UNIX socket transport module. [ 1.087319] RPC: Registered udp transport module. [ 1.089049] RPC: Registered tcp transport module. [ 1.090713] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.093129] NET: Registered protocol family 44 [ 1.094800] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.096956] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.099166] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.101705] PCI: CLS 0 bytes, default 64 [ 1.104315] Unpacking initramfs... [ 2.662756] debug: unmapping init [mem 0xffff912cfcc54000-0xffff912cfffbffff] [ 2.666688] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.669098] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.672173] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.191383] Initialise system trusted keyrings [ 3.193102] Key type blacklist registered [ 3.195000] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.203704] zbud: loaded [ 3.206553] *** VALIDATE nfs *** [ 3.207829] *** VALIDATE nfs4 *** [ 3.209412] pstore: using deflate compression [ 3.213179] Platform Keyring initialized [ 3.309759] NET: Registered protocol family 38 [ 3.313381] Key type asymmetric registered [ 3.314814] Asymmetric key parser 'x509' registered [ 3.316718] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.320125] io scheduler mq-deadline registered [ 3.321700] io scheduler kyber registered [ 3.323353] io scheduler bfq registered [ 3.325697] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.328872] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.331741] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.334705] ACPI: Power Button [PWRF] [ 3.339465] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.346474] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.358553] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.364927] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.378044] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.407705] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.438024] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.442787] Non-volatile memory driver v1.3 [ 3.444596] Linux agpgart interface v0.103 [ 3.479458] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.482746] vda: detected capacity change from 0 to 74702848 [ 3.591283] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.594847] vdb: detected capacity change from 0 to 1073741824 [ 3.614239] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.616641] vdc: detected capacity change from 0 to 2621440000 [ 3.630384] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.633132] vdd: detected capacity change from 0 to 2621440000 [ 3.648161] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.650505] vde: detected capacity change from 0 to 4294967296 [ 3.665522] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.667851] vdf: detected capacity change from 0 to 4294967296 [ 3.674337] libphy: Fixed MDIO Bus: probed [ 3.681163] usbcore: registered new interface driver usbserial_generic [ 3.683653] usbserial: USB Serial support registered for generic [ 3.686684] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.691391] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.693195] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.695729] mousedev: PS/2 mouse device common for all mice [ 3.702145] rtc_cmos 00:05: RTC can wake from S4 [ 3.705736] rtc_cmos 00:05: registered as rtc0 [ 3.707105] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.709717] intel_pstate: CPU model not supported [ 3.711813] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.715712] hid: raw HID events driver (C) Jiri Kosina [ 3.717314] usbcore: registered new interface driver usbhid [ 3.718907] usbhid: USB HID core driver [ 3.720148] drop_monitor: Initializing network drop monitor service [ 3.722533] Initializing XFRM netlink socket [ 3.724346] NET: Registered protocol family 10 [ 3.727925] Segment Routing with IPv6 [ 3.729513] NET: Registered protocol family 17 [ 3.730914] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.732631] mpls_gso: MPLS GSO support [ 3.741945] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.745822] RAS: Correctable Errors collector initialized. [ 3.748163] AVX version of gcm_enc/dec engaged. [ 3.750045] AES CTR mode by8 optimization enabled [ 3.835084] sched_clock: Marking stable (3835063264, 0)->(4718850814, -883787550) [ 3.837965] registered taskstats version 1 [ 3.839627] Loading compiled-in X.509 certificates [ 3.841870] zswap: loaded using pool lzo/zbud [ 3.867850] Key type big_key registered [ 3.879847] Key type encrypted registered [ 3.881182] ima: No TPM chip found, activating TPM-bypass! [ 3.883410] ima: Allocated hash algorithm: sha1 [ 3.885119] ima: No architecture policies found [ 3.886672] evm: Initialising EVM extended attributes: [ 3.888338] evm: security.selinux [ 3.889448] evm: security.ima [ 3.890554] evm: security.capability [ 3.891760] evm: HMAC attrs: 0x1 [ 3.894244] rtc_cmos 00:05: setting system clock to 2026-08-13 13:34:42 UTC (1786628082) [ 3.901321] debug: unmapping init [mem 0xffffffffb9403000-0xffffffffb95fffff] [ 3.904153] debug: unmapping init [mem 0xffffffffb8182000-0xffffffffb8458fff] [ 3.912093] Write protecting the kernel read-only data: 28672k [ 3.915811] debug: unmapping init [mem 0xffffffffb6803000-0xffffffffb69fffff] [ 3.918497] debug: unmapping init [mem 0xffffffffb7114000-0xffffffffb71fffff] [ 3.949828] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.956690] systemd[1]: Detected virtualization kvm. [ 3.958260] systemd[1]: Detected architecture x86-64. [ 3.959796] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.985281] systemd[1]: No hostname configured. [ 3.986763] systemd[1]: Set hostname to . [ 3.988559] random: systemd: uninitialized urandom read (16 bytes read) [ 3.990939] systemd[1]: Initializing machine ID from random generator. [ 4.024093] random: ln: uninitialized urandom read (6 bytes read) [ 4.152821] random: systemd: uninitialized urandom read (16 bytes read) [ 4.156759] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.187444] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.198575] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.872961] device-mapper: uevent: version 1.0.3 [ 4.874664] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.649656] random: fast init done [ 5.654434] virtio_net virtio0 ens2: renamed from eth0 [ 5.758712] scsi host0: ata_piix [ 5.783869] scsi host1: ata_piix [ 5.785377] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.787916] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.869394] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.397877] random: crng init done [ 10.399104] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.297649] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.624674] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.881876] SELinux: Disabled at runtime. [ 12.947883] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.956260] systemd[1]: Detected virtualization kvm. [ 12.958273] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.638556] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.644246] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.654602] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.661947] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.665585] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.679171] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.685977] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK [0[ 13.887827] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 14.392102] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.743570] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.833712] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.008275] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 15.077580] EDAC sbridge: Ver: 1.1.2 [ 16.889179] Key type dns_resolver registered [ 17.198885] NFS: Registering the id_resolver key type [ 17.200719] Key type id_resolver registered [ 17.202225] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [** ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg105-server login: [ 52.984312] libcfs: loading out-of-tree module taints kernel. [ 53.032414] Key type ._llcrypt registered [ 53.035735] Key type .llcrypt registered [ 53.160329] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 75.965612] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 77.683570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 77.702329] alg: No test for adler32 (adler32-zlib) [ 79.414750] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 80.117614] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 81.921259] Key type lgssc registered [ 84.204394] Lustre: Echo OBD driver; http://www.lustre.org/ [ 101.729972] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 146.750733] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 162.140595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 162.172066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 163.503044] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 163.526260] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 163.661102] Lustre: lustre-MDT0000: new disk, initializing [ 163.840622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 163.868644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 169.138195] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.281875] hrtimer: interrupt took 10629908 ns [ 185.161605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 185.287148] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 185.312437] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 185.316432] Lustre: Skipped 1 previous similar message [ 185.394529] Lustre: lustre-MDT0001: new disk, initializing [ 185.468576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 185.490853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 185.500148] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 190.930801] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.799260] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 206.136866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 206.568514] Lustre: lustre-OST0000: new disk, initializing [ 206.572615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 206.578613] Lustre: 8446:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 206.736708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 212.035269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 212.059448] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 212.116342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 212.404380] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.272465] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 229.344585] Lustre: lustre-OST0001: new disk, initializing [ 229.346982] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 229.351683] Lustre: 9519:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 229.419274] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 236.426743] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.152748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 238.164644] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 238.284907] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 249.830216] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 257.709213] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 264.295523] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing check_logdir /tmp/testlogs/ [ 269.496067] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing yml_node [ 273.151728] Lustre: DEBUG MARKER: Client: 2.17.57.2 [ 275.803550] Lustre: DEBUG MARKER: MDS: 2.17.57.2 [ 279.149917] Lustre: DEBUG MARKER: OSS: 2.17.57.2 [ 281.452365] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 13 09:39:16 EDT 2026 [ 299.830595] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 301.604732] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 320.484322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 320.487546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 320.508592] Lustre: Skipped 3 previous similar messages [ 320.523932] Lustre: Skipped 3 previous similar messages [ 325.600770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 325.614855] Lustre: Skipped 3 previous similar messages [ 326.460506] Lustre: server umount lustre-MDT0000 complete [ 330.540464] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628409 with bad export cookie 1233357211157796887 [ 330.565126] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 330.719958] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.724500] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 330.731367] Lustre: Skipped 1 previous similar message [ 330.740756] Lustre: Skipped 1 previous similar message [ 335.840439] LustreError: 6517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 335.840716] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 335.860102] LustreError: 6517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 335.885164] Lustre: Skipped 1 previous similar message [ 337.010813] Lustre: server umount lustre-MDT0001 complete [ 341.298830] Lustre: server umount lustre-OST0000 complete [ 345.887085] Lustre: server umount lustre-OST0001 complete [ 354.645228] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 364.774184] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 413.811055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 414.097894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 414.134368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 414.245562] Lustre: lustre-MDT0000: new disk, initializing [ 414.349515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 414.395890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 420.495576] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 432.605303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 432.769398] Lustre: 15073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 432.826595] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 432.834393] Lustre: Skipped 1 previous similar message [ 432.978046] Lustre: lustre-MDT0001: new disk, initializing [ 433.180437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 433.240990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 433.267930] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 438.109535] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 449.739627] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 456.933790] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 464.890529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 465.324418] Lustre: lustre-OST0000: new disk, initializing [ 465.328503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 465.334200] Lustre: 17045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 465.457412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.491706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 467.522205] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 467.717779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 472.871996] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 484.183873] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 490.149109] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 490.321451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 496.384927] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 496.708526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 503.782438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 503.786571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 503.799904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 503.809422] Lustre: Skipped 1 previous similar message [ 503.834359] Lustre: Skipped 1 previous similar message [ 505.458195] Lustre: server umount lustre-OST0000 complete [ 509.925593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 509.934204] LustreError: Skipped 1 previous similar message [ 509.937905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 514.019240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 514.025851] Lustre: Skipped 1 previous similar message [ 514.046261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.078475] Lustre: server umount lustre-MDT0000 complete [ 519.141151] LustreError: 17022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 519.162058] LustreError: 17022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 520.474698] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628599 with bad export cookie 1233357211157798420 [ 520.477696] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 520.484177] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 521.162081] Lustre: server umount lustre-MDT0001 complete [ 527.199901] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 09:43:23 (1786628603) [ 549.037802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 549.670614] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.783523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 554.981139] LustreError: 19922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 555.073841] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 560.110404] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.714222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.985827] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 578.373199] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 584.786577] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 587.390937] Lustre: Failing over lustre-MDT0000 [ 587.771519] Lustre: server umount lustre-MDT0000 complete [ 589.472486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.474944] LustreError: 19922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 589.487689] Lustre: Skipped 1 previous similar message [ 606.180523] Lustre: 3641:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786628668/real 1786628668] req@ffff912d76f49c00 x1873415407565568/t0(0) o400->MGC192.168.201.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786628684 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 606.212236] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 606.225794] LustreError: 19917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 606.257327] LustreError: 19917:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 608.085793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 615.771490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 615.785716] Lustre: Skipped 1 previous similar message [ 615.898860] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 621.038344] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 621.047742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 621.073936] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 621.170579] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 632.015340] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 639.458713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 639.466343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 639.474327] Lustre: Skipped 1 previous similar message [ 639.491567] Lustre: Skipped 2 previous similar messages [ 643.005491] Lustre: server umount lustre-MDT0000 complete [ 644.576291] LustreError: 20649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 644.613374] LustreError: 20649:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 647.188569] LustreError: 19903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628725 with bad export cookie 1233357211157799834 [ 647.198664] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 647.199781] LustreError: 19903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 647.621821] Lustre: server umount lustre-MDT0001 complete [ 657.738159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 663.190244] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 671.763951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 677.188920] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 685.438538] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 692.647484] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 696.068098] Lustre: Failing over lustre-MDT0000 [ 696.469478] Lustre: server umount lustre-MDT0000 complete [ 697.140488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 697.156269] Lustre: Skipped 1 previous similar message [ 712.671177] Lustre: 3640:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786628775/real 1786628775] req@ffff912d74ac0000 x1873415407595904/t0(0) o400->MGC192.168.201.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786628791 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 712.674714] LustreError: 23260:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 712.723512] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 712.763529] LustreError: 23260:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 716.157841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 724.307615] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 729.581040] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 729.611390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 729.621332] Lustre: Skipped 1 previous similar message [ 729.637539] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 729.642140] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 739.942798] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 744.930891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 744.953962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 759.263355] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 759.651659] Lustre: server umount lustre-MDT0000 complete [ 763.812723] LustreError: 23241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628842 with bad export cookie 1233357211157800772 [ 763.820573] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 763.824455] LustreError: 23241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 764.250362] Lustre: server umount lustre-MDT0001 complete [ 775.018595] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:47:30 (1786628850) [ 794.104708] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 805.835466] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 849.772852] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 862.892268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 863.174791] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 863.210301] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 863.324791] Lustre: lustre-MDT0000: new disk, initializing [ 863.459303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 863.466031] Lustre: Skipped 3 previous similar messages [ 863.481554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 869.180914] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 880.855251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 880.953819] Lustre: 29635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 881.433050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 886.878752] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 897.329122] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 904.479546] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 912.288810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 912.737405] Lustre: 31608:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 914.422522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 914.472900] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 914.609401] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 914.628129] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 920.325739] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 930.283637] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 936.264781] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 958.472906] Lustre: server umount lustre-OST0000 complete [ 963.562567] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 963.572951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 963.583421] Lustre: Skipped 2 previous similar messages [ 963.589059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 963.595045] Lustre: Skipped 6 previous similar messages [ 973.521964] Lustre: server umount lustre-MDT0000 complete [ 976.353561] LustreError: 29644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 976.367306] LustreError: 29644:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 977.376133] Lustre: 3643:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786629039/real 1786629039] req@ffff912c49111c00 x1873415407660544/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786629055 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [ 978.263813] LustreError: 29628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629056 with bad export cookie 1233357211157801171 [ 978.268733] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 978.271237] LustreError: 29628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 978.607295] Lustre: server umount lustre-MDT0001 complete [ 991.482948] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 995.113894] Key type lgssc unregistered [ 995.576139] LNet: 33591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.582698] LNetError: 33591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 995.605858] LNet: Removed LNI 192.168.201.105@tcp [ 996.875392] Key type .llcrypt unregistered [ 996.878238] Key type ._llcrypt unregistered [ 1025.880161] Key type ._llcrypt registered [ 1025.881709] Key type .llcrypt registered [ 1026.025463] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1027.654241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1027.681135] alg: No test for adler32 (adler32-zlib) [ 1028.952584] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 1029.308573] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1031.031295] Key type lgssc registered [ 1032.299738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1040.474752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1040.505854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1042.162959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1047.637141] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1051.638781] LustreError: 35006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1051.688924] LustreError: 35006:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1056.740060] LustreError: 35005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1059.109871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1059.446927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1065.594486] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1074.546980] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1081.436361] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1089.887970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1090.458334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1098.328915] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1103.815029] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1106.874141] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1111.998990] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1126.539485] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:53:22 (1786629202) [ 1128.545633] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1130.533981] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1137.201077] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:53:32 (1786629212) [ 1146.475319] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:53:42 (1786629222) [ 1147.793066] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1158.321845] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1158.327403] Lustre: Skipped 2 previous similar messages [ 1160.738326] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:53:56 (1786629236) [ 1163.276127] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1163.280434] Lustre: Skipped 2 previous similar messages [ 1172.292537] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1172.296906] Lustre: Skipped 4 previous similar messages [ 1187.521106] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:54:23 (1786629263) [ 1190.535423] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1194.757521] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1204.062508] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:54:40 (1786629280) [ 1206.489423] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1206.493691] Lustre: Skipped 6 previous similar messages [ 1224.020139] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:54:59 (1786629299) [ 1242.080166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1242.083217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1242.103163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1245.154374] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1245.156813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1245.167536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1245.199054] Lustre: Skipped 1 previous similar message [ 1246.879507] Lustre: server umount lustre-OST0000 complete [ 1253.857538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1253.865210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1253.895895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1257.785875] Lustre: server umount lustre-MDT0000 complete [ 1260.514649] LustreError: 35002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1260.538295] LustreError: 35002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1261.962366] LustreError: 34985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629340 with bad export cookie 15776602375483541845 [ 1261.967464] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1261.975213] LustreError: 34985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1262.608788] Lustre: server umount lustre-MDT0001 complete [ 1276.061560] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1279.593106] Key type lgssc unregistered [ 1280.001985] LNet: 42867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1280.014937] LNetError: 42867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1280.042203] LNet: Removed LNI 192.168.201.105@tcp [ 1281.263547] Key type .llcrypt unregistered [ 1281.265371] Key type ._llcrypt unregistered [ 1301.705784] Key type ._llcrypt registered [ 1301.708728] Key type .llcrypt registered [ 1301.846316] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 1318.761391] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1320.078711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1320.111432] alg: No test for adler32 (adler32-zlib) [ 1321.085490] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 1321.299451] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1322.983418] Key type lgssc registered [ 1324.230735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1362.501372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1362.526886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1363.894107] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1363.954194] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1364.097610] Lustre: lustre-MDT0000: new disk, initializing [ 1364.226150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1364.262057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1369.768368] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1381.743870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1381.954473] Lustre: 46021:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1382.085928] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1382.097282] Lustre: Skipped 1 previous similar message [ 1382.214784] Lustre: lustre-MDT0001: new disk, initializing [ 1382.314676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1382.351063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1382.369746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1387.898860] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1398.330308] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1404.855622] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1414.398045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1414.898163] Lustre: lustre-OST0000: new disk, initializing [ 1414.902900] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1414.911309] Lustre: 47994:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1415.091754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1420.432614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1420.442659] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1420.664430] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1423.039099] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1432.206817] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1438.504699] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1438.863677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1443.821974] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1443.977596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1446.368394] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1446.373717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.374602] LustreError: Skipped 1 previous similar message [ 1446.392515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1451.501585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1451.513184] Lustre: Skipped 2 previous similar messages [ 1456.616698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1456.625914] Lustre: Skipped 1 previous similar message [ 1460.192685] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1460.525368] Lustre: server umount lustre-OST0000 complete [ 1466.852581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1466.867381] Lustre: Skipped 1 previous similar message [ 1466.871973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1466.875540] Lustre: Skipped 1 previous similar message [ 1471.971430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1471.985190] Lustre: Skipped 1 previous similar message [ 1472.696462] Lustre: server umount lustre-MDT0000 complete [ 1478.918334] LustreError: 46016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629557 with bad export cookie 11743649565727603490 [ 1478.930333] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1478.937545] LustreError: 46016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1479.329880] Lustre: server umount lustre-MDT0001 complete [ 1491.468467] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:59:27 (1786629567) [ 1500.597068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1501.174649] LustreError: 50456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1501.188495] LustreError: 50456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1501.239917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1506.302419] LustreError: 50457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1507.060615] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1510.434835] LustreError: 50456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1515.490905] LustreError: 50457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1517.784314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1518.450064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1524.190434] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1533.706300] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1541.625122] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1552.545089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1553.141089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1560.882674] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1569.848477] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1579.179727] LustreError: 52987:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1580.210350] Lustre: Modifying parameter general.timeout=1 in log params [ 1581.238168] Lustre: Modifying parameter general.timeout=2 in log params [ 1583.301082] Lustre: Modifying parameter general.timeout=4 in log params [ 1583.304400] Lustre: Skipped 1 previous similar message [ 1587.974087] Lustre: Modifying parameter general.timeout=9 in log params [ 1587.979764] Lustre: Skipped 4 previous similar messages [ 1596.562628] Lustre: Modifying parameter general.timeout=18 in log params [ 1596.564690] Lustre: Skipped 8 previous similar messages [ 1612.726328] Lustre: Modifying parameter general.timeout=34 in log params [ 1612.731505] Lustre: Skipped 15 previous similar messages [ 1645.076549] Lustre: Modifying parameter general.timeout=62 in log params [ 1645.079293] Lustre: Skipped 27 previous similar messages [ 1709.500203] Lustre: Modifying parameter general.timeout=120 in log params [ 1709.510223] Lustre: Skipped 57 previous similar messages [ 1731.647774] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 10:03:27 (1786629807) [ 1734.020169] Lustre: 59955:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1735.610208] Lustre: 60053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1737.866526] Lustre: 60166:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1742.954281] Lustre: 60362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1742.960358] Lustre: 60362:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1776.116307] Lustre: 61836:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1776.126551] Lustre: 61836:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1776.165697] LustreError: 61836:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1781.925633] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 10:04:16 (1786629856) [ 1800.167586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1800.177727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.184620] Lustre: Skipped 1 previous similar message [ 1800.187913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1803.806189] Lustre: server umount lustre-MDT0000 complete [ 1805.798278] LustreError: 50456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1805.808941] LustreError: 50456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1807.660736] LustreError: 50437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629886 with bad export cookie 11743649565727604372 [ 1807.661924] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1807.673268] LustreError: 50437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1808.169572] Lustre: server umount lustre-MDT0001 complete [ 1823.275387] Lustre: server umount lustre-OST0000 complete [ 1853.707095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1853.994856] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1854.012827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1854.016832] Lustre: Skipped 32 previous similar messages [ 1854.241337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1859.210645] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1868.440616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1868.544971] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1868.566798] Lustre: 64372:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1873.751535] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1883.607352] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1889.821586] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1898.197344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1898.464812] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1898.908968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1898.914910] Lustre: Skipped 1 previous similar message [ 1907.189365] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1915.465568] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1917.804027] Lustre: 66938:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1925.604983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1925.607624] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1925.612551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1925.612562] Lustre: Skipped 2 previous similar messages [ 1925.617127] Lustre: Skipped 3 previous similar messages [ 1930.687338] Lustre: server umount lustre-OST0000 complete [ 1935.329126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1935.339310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1941.111715] Lustre: server umount lustre-MDT0000 complete [ 1945.413179] LustreError: 64363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630024 with bad export cookie 11743649565727607543 [ 1945.420037] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1945.425031] LustreError: 64363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1945.862191] Lustre: server umount lustre-MDT0001 complete [ 1959.148600] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1962.571417] Key type lgssc unregistered [ 1963.014806] LNet: 68150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1963.032934] LNetError: 68150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1963.056090] LNet: Removed LNI 192.168.201.105@tcp [ 1964.388469] Key type .llcrypt unregistered [ 1964.390559] Key type ._llcrypt unregistered [ 1993.300994] Key type ._llcrypt registered [ 1993.302744] Key type .llcrypt registered [ 1993.405739] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1994.770752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1994.916396] alg: No test for adler32 (adler32-zlib) [ 1995.964417] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 1996.202448] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1997.863301] Key type lgssc registered [ 1999.170769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2008.009301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2008.026544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2009.630861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2014.029633] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2019.814358] LustreError: 69566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2019.828480] LustreError: 69566:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2022.191548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2022.649891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2027.846813] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2038.834423] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2046.488467] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2054.517953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2055.342333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2062.997314] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2071.636369] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2089.081837] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2099.175587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2099.186747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2099.203848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2100.704974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2100.716951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2104.994212] Lustre: server umount lustre-MDT0000 complete [ 2105.831437] LustreError: 70560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.855314] LustreError: 70560:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2109.545865] LustreError: 69547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630188 with bad export cookie 14022203786131224894 [ 2109.554700] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2109.565778] LustreError: 69547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2110.097319] Lustre: server umount lustre-MDT0001 complete [ 2125.062712] Lustre: server umount lustre-OST0000 complete [ 2157.347703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2157.481244] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2157.493283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2157.648225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2163.741906] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2175.707461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2175.787951] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2175.812546] Lustre: 74466:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2176.378391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2182.384723] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2191.434750] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2198.811903] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2209.196885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2209.348469] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2209.726569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2219.799307] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2229.802337] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2231.796581] Lustre: 77045:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2234.413604] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2241.504262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2241.511449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2241.512246] LustreError: Skipped 1 previous similar message [ 2241.513524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2241.513532] Lustre: Skipped 2 previous similar messages [ 2241.530109] Lustre: Skipped 2 previous similar messages [ 2246.278799] Lustre: server umount lustre-OST0000 complete [ 2251.322841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2251.352890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2251.353658] Lustre: Skipped 1 previous similar message [ 2251.372341] Lustre: Skipped 2 previous similar messages [ 2256.352339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2256.360698] Lustre: Skipped 1 previous similar message [ 2257.783358] Lustre: server umount lustre-MDT0000 complete [ 2261.483855] LustreError: 77019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2261.514492] LustreError: 77019:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2263.276334] LustreError: 74457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630341 with bad export cookie 14022203786131225734 [ 2263.287985] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2263.297662] LustreError: 74457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2263.809230] Lustre: server umount lustre-MDT0001 complete [ 2280.147715] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2284.077191] Key type lgssc unregistered [ 2284.568132] LNet: 78260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2284.585298] LNetError: 78260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2284.604170] LNet: Removed LNI 192.168.201.105@tcp [ 2286.014699] Key type .llcrypt unregistered [ 2286.019948] Key type ._llcrypt unregistered [ 2315.746172] Key type ._llcrypt registered [ 2315.750172] Key type .llcrypt registered [ 2315.936651] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2316.948964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2317.006620] alg: No test for adler32 (adler32-zlib) [ 2318.317894] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 2318.561884] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2320.311176] Key type lgssc registered [ 2321.639801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2330.760091] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2330.790186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2332.671086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2338.579691] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2342.885238] LustreError: 79672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2342.909240] LustreError: 79672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2348.003599] LustreError: 79673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2348.583639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2348.942644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2354.560928] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2365.567738] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2373.340173] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2380.290642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2380.946547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2387.994712] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2397.494609] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2408.083238] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2409.496340] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2410.703978] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2412.892458] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2412.896789] Lustre: Skipped 1 previous similar message [ 2417.659617] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2417.662296] Lustre: Skipped 3 previous similar messages [ 2426.312264] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2426.324910] Lustre: Skipped 6 previous similar messages [ 2443.499475] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2443.503356] Lustre: Skipped 15 previous similar messages [ 2475.538150] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2475.544169] Lustre: Skipped 28 previous similar messages [ 2533.017676] Lustre: 87792:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2535.949714] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 10:16:51 (1786630611) [ 2537.906949] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2540.287178] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 10:16:55 (1786630615) [ 2546.150943] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2546.152824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2546.166717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2546.209069] Lustre: Skipped 1 previous similar message [ 2550.267223] Lustre: server umount lustre-OST0000 complete [ 2556.384970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2556.392595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2556.394108] Lustre: Skipped 1 previous similar message [ 2556.411832] Lustre: Skipped 2 previous similar messages [ 2561.527664] LustreError: 79667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2561.755911] Lustre: server umount lustre-MDT0000 complete [ 2565.681768] LustreError: 79673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.720145] LustreError: 79673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2566.117188] LustreError: 82912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630644 with bad export cookie 12798418885089033727 [ 2566.121989] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2566.134648] LustreError: 82912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2566.622291] Lustre: server umount lustre-MDT0001 complete [ 2581.914424] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2585.664931] Key type lgssc unregistered [ 2586.031900] LNet: 89044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2586.044525] LNetError: 89044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2586.068653] LNet: Removed LNI 192.168.201.105@tcp [ 2587.482420] Key type .llcrypt unregistered [ 2587.484082] Key type ._llcrypt unregistered [ 2593.591967] Key type ._llcrypt registered [ 2593.599524] Key type .llcrypt registered [ 2593.686109] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2599.768084] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules [ 2601.141577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2601.422628] alg: No test for adler32 (adler32-zlib) [ 2602.662876] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 2603.142374] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2604.847121] Key type lgssc registered [ 2606.355481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2608.182807] LustreError: 90296:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2631.242993] LustreError: 90296:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2631.963387] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2649.084743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2649.134274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2650.827916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2657.316773] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2661.346765] LustreError: 91322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2661.373685] LustreError: 91322:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2666.515537] LustreError: 91321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2667.807486] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 10:19:03 (1786630743) [ 2671.584585] LustreError: 91322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2674.807071] LustreError: 91353:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2674.819324] LustreError: 91353:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2674.833947] LustreError: 91353:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 24, retries 0, failed: rc = -5 [ 2675.584821] Lustre: server umount lustre-MDT0000 complete [ 2694.928490] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2698.326946] Key type lgssc unregistered [ 2698.655400] LNet: 92852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2698.658767] LNetError: 92852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2698.672758] LNet: Removed LNI 192.168.201.105@tcp [ 2699.457512] Key type .llcrypt unregistered [ 2699.459205] Key type ._llcrypt unregistered [ 2720.152157] Key type ._llcrypt registered [ 2720.157702] Key type .llcrypt registered [ 2720.350554] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2722.285713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2722.342219] alg: No test for adler32 (adler32-zlib) [ 2723.475347] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 2723.835421] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2725.567212] Key type lgssc registered [ 2727.252502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2736.860368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2736.902669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2738.678722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2743.905094] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2748.898641] LustreError: 94075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2748.932339] LustreError: 94075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2754.023644] LustreError: 94074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2754.429286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2755.601428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2761.462384] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2770.329882] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2776.267593] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2784.694344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2785.157801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2791.182199] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2800.559942] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2861.837601] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 10:22:17 (1786630937) [ 2863.582497] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2865.463727] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 10:22:21 (1786630941) [ 2870.243625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2870.244438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2870.263243] Lustre: Skipped 2 previous similar messages [ 2870.277558] Lustre: Skipped 2 previous similar messages [ 2875.364226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2875.372174] Lustre: Skipped 2 previous similar messages [ 2876.556146] Lustre: server umount lustre-MDT0000 complete [ 2880.483167] LustreError: 94074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2880.514605] LustreError: 94074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2880.722939] LustreError: 94056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630959 with bad export cookie 7572504391361789256 [ 2880.731816] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2880.733964] LustreError: 94056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2881.052126] Lustre: server umount lustre-MDT0001 complete [ 2896.107433] Lustre: server umount lustre-OST0000 complete [ 2909.715253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2910.414330] LustreError: 98386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2910.598122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2915.810046] LustreError: 98387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2916.395899] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2926.050912] LustreError: 98386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2926.082413] LustreError: 98386:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2927.154938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2928.071227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2934.208171] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2943.122324] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2950.334547] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2965.426987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2965.649010] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2965.654656] LustreError: 98375:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2965.681705] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2965.693822] LustreError: 100480:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2965.707273] LustreError: 100480:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2965.715621] LustreError: 100480:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2965.817661] Lustre: server umount lustre-OST0000 complete [ 2965.832961] LustreError: 100480:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2975.688355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2975.976091] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2975.983103] LustreError: 98375:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2976.002604] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2976.011895] LustreError: 100946:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2976.020996] LustreError: 100946:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2976.026298] LustreError: 100946:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2976.143387] Lustre: server umount lustre-OST0000 complete [ 2976.149542] LustreError: 100946:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2987.273404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2987.576510] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2987.587743] Lustre: Found index 0 for lustre-OST0000, updating log [ 2987.599318] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2987.709606] Lustre: lustre-OST0000: new disk, initializing [ 2987.716393] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2987.724189] Lustre: 101482:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2988.058824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2995.812833] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3000.351614] Lustre: Failing over lustre-OST0000 [ 3000.608582] Lustre: server umount lustre-OST0000 complete [ 3005.450820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3005.459868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3005.465320] Lustre: Skipped 1 previous similar message [ 3010.530389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3010.533876] Lustre: Skipped 1 previous similar message [ 3011.892307] Lustre: server umount lustre-MDT0000 complete [ 3015.595340] LustreError: 100499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631094 with bad export cookie 7572504391361792000 [ 3015.606637] LustreError: 100499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3015.607306] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3026.061501] Lustre: server umount lustre-MDT0001 complete [ 3035.588789] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 10:25:11 (1786631111) [ 3060.215778] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3074.193865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3074.779024] LustreError: 103853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3074.928513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3079.597445] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3088.884852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3094.270260] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3097.087490] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3104.543893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3105.456274] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3113.433467] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3113.639798] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3113.706915] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3113.708439] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3125.577240] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3125.796340] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3126.017624] Lustre: lustre-OST0001: new disk, initializing [ 3126.023406] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3126.043994] Lustre: 106172:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3126.619082] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3126.623302] Lustre: Skipped 2 previous similar messages [ 3134.138084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3134.161833] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3134.328048] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3135.488435] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3145.034952] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3155.444893] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3161.057771] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3161.074825] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3161.100244] Lustre: Skipped 1 previous similar message [ 3161.115893] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3165.178460] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3165.203224] Lustre: Skipped 1 previous similar message [ 3166.829838] Lustre: server umount lustre-MDT0001 complete [ 3169.578420] LustreError: 105370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3169.591550] LustreError: 105370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3177.686309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3177.796694] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3177.801267] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3177.914065] Lustre: 103842:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3179.507751] LustreError: 107744:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3179.511471] LustreError: 107744:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3179.527865] LustreError: 103845:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3179.578376] LustreError: 107756:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.105@tcp: cfg command failed: rc = -17 [ 3179.582322] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3179.582322] [ 3183.622368] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3184.051710] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3196.625843] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 10:27:52 (1786631272) [ 3216.580042] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3241.690527] Lustre: 109787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3266.961567] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3271.329417] Lustre: 110922:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3371.737614] Lustre: server umount lustre-MDT0000 complete [ 3373.024168] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3373.032377] LustreError: 103850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3373.047634] Lustre: Skipped 3 previous similar messages [ 3373.059256] LustreError: 103850:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 3376.587328] LustreError: 110924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631455 with bad export cookie 7572504391361793001 [ 3376.594111] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3376.599559] LustreError: 110924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3376.992300] Lustre: server umount lustre-MDT0001 complete [ 3390.697882] Lustre: server umount lustre-OST0000 complete [ 3394.407366] Lustre: 93385:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786631456/real 1786631456] req@ffff912d76b0d180 x1873418180220160/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786631472 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3394.444450] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3395.014081] Lustre: server umount lustre-OST0001 complete [ 3401.836253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3413.669397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3423.743108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3435.843566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3461.434597] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3475.928859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3475.954240] Lustre: lustre-MDT0000: reset Object Index mappings [ 3476.702410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3476.705136] Lustre: Skipped 1 previous similar message [ 3482.744487] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3494.065428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3494.077964] Lustre: lustre-MDT0001: reset Object Index mappings [ 3500.577391] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3504.706515] Lustre: 116026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3512.450329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3519.478287] LustreError: 116380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3519.499613] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 3519.509509] LustreError: 116380:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3520.310551] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3521.521325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3530.251590] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3536.387287] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3536.417847] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3538.854577] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3549.194323] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3555.288934] Lustre: 117904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3615.439963] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 10:34:50 (1786631690) [ 3642.850838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3642.859843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3642.873167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3642.878946] Lustre: Skipped 4 previous similar messages [ 3646.379467] Lustre: server umount lustre-MDT0000 complete [ 3651.163034] LustreError: 114860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631729 with bad export cookie 7572504391361839873 [ 3651.167969] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3651.178884] LustreError: 114860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3658.301168] Lustre: server umount lustre-MDT0001 complete [ 3669.159112] Lustre: server umount lustre-OST0000 complete [ 3679.731539] Lustre: server umount lustre-OST0001 complete [ 3687.121647] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 3700.254433] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3748.482744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3748.767395] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3748.815235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3748.946771] Lustre: lustre-MDT0000: new disk, initializing [ 3749.145762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3749.160141] Lustre: Skipped 3 previous similar messages [ 3749.226701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3755.352579] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3764.162871] Lustre: server umount lustre-MDT0000 complete [ 3777.341805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3777.785134] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3777.817172] Lustre: 122179:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3777.849487] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3783.114628] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3788.914151] Lustre: server umount lustre-MDT0000 complete [ 3799.822881] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 10:37:55 (1786631875) [ 3801.639461] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3803.502768] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 10:37:59 (1786631879) [ 3824.135408] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3836.170581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3841.923130] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3852.446523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3852.740508] Lustre: 124229:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3852.833751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3852.837318] Lustre: Skipped 1 previous similar message [ 3852.952373] Lustre: lustre-MDT0001: new disk, initializing [ 3853.150018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3853.172712] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3859.248784] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3865.159466] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3873.856233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3874.154426] Lustre: lustre-OST0000: new disk, initializing [ 3874.158872] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3874.175985] Lustre: 125807:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3875.344074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3875.360576] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3875.472811] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3882.317896] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3896.665547] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3897.016241] Lustre: lustre-OST0001: new disk, initializing [ 3897.035382] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3897.049611] Lustre: 126679:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3899.046804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3899.065842] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3899.333474] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3907.440236] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3918.217183] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3923.260168] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4366.733851] Lustre: 133107:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4366.741300] Lustre: 133107:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4366.745929] Lustre: 133107:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4366.755781] Lustre: 133107:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4366.776787] Lustre: 133107:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4366.784793] Lustre: 133107:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4707.685858] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:53:01 (1786632781) [ 4723.692263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4723.703936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4723.714806] Lustre: Skipped 5 previous similar messages [ 4723.721309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4723.726496] Lustre: Skipped 8 previous similar messages [ 4729.826949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4729.840292] Lustre: Skipped 6 previous similar messages [ 4730.635932] Lustre: server umount lustre-MDT0000 complete [ 4734.950441] LustreError: 127510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4734.980857] LustreError: 127510:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 4736.397889] LustreError: 124219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786632815 with bad export cookie 7572504391361909873 [ 4736.400994] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4736.425996] LustreError: 124219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4740.081650] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4740.105459] Lustre: Skipped 3 previous similar messages [ 4740.128203] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4740.131067] Lustre: Skipped 1 previous similar message [ 4742.964486] Lustre: server umount lustre-MDT0001 complete [ 4754.488421] Lustre: server umount lustre-OST0000 complete [ 4765.591554] Lustre: server umount lustre-OST0001 complete [ 4774.504239] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 4785.517520] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4826.107660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4826.352640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4826.386889] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4826.466271] Lustre: lustre-MDT0000: new disk, initializing [ 4826.561228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4826.564812] Lustre: Skipped 5 previous similar messages [ 4826.575984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4831.856646] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4842.492649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4842.704097] Lustre: 136172:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4842.831882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4842.839534] Lustre: Skipped 1 previous similar message [ 4843.014709] Lustre: lustre-MDT0001: new disk, initializing [ 4843.184307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4843.192969] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4848.195452] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4859.919613] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4866.232824] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4873.877104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4874.403238] Lustre: lustre-OST0000: new disk, initializing [ 4874.409238] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4874.417850] Lustre: 138140:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4874.560906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4874.566767] Lustre: Skipped 1 previous similar message [ 4875.846186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4875.862530] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4876.100143] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4881.776664] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4891.796196] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4902.375353] Lustre: lustre-MDD0000: changelog on [ 4905.597566] Lustre: lustre-MDD0001: changelog on [ 6689.221384] Lustre: lustre-MDD0001: changelog off [ 6691.182071] Lustre: 137049:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6692.125965] Lustre: lustre-MDD0000: changelog off [ 6698.985617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6698.990477] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6699.004951] Lustre: Skipped 1 previous similar message [ 6699.018929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6699.026970] Lustre: Skipped 3 previous similar messages [ 6704.100666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6704.108145] Lustre: Skipped 1 previous similar message [ 6704.576670] Lustre: server umount lustre-OST0000 complete [ 6709.216975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6709.231280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6709.234160] Lustre: Skipped 2 previous similar messages [ 6709.248383] Lustre: Skipped 1 previous similar message [ 6714.336247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6714.803183] Lustre: server umount lustre-MDT0000 complete [ 6719.348703] LustreError: 138139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634797 with bad export cookie 7572504391362734081 [ 6719.352264] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6719.359878] LustreError: 138139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6719.766547] Lustre: server umount lustre-MDT0001 complete [ 6734.012425] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 6737.345355] Key type lgssc unregistered [ 6737.778275] LNet: 147266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6737.800313] LNetError: 147266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6738.859604] LNet: Removed LNI 192.168.201.105@tcp [ 6739.967834] Key type .llcrypt unregistered [ 6739.979547] Key type ._llcrypt unregistered [ 6760.662532] Key type ._llcrypt registered [ 6760.671405] Key type .llcrypt registered [ 6760.842776] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 6777.033616] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6778.891028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6778.921665] alg: No test for adler32 (adler32-zlib) [ 6780.068800] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 6780.337930] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 6782.041176] Key type lgssc registered [ 6783.198967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6818.700386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6818.722203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6820.062682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6820.113321] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6820.221236] Lustre: lustre-MDT0000: new disk, initializing [ 6820.360695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6820.381935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6825.239840] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6836.750707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6836.880546] Lustre: 150471:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6836.956529] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6836.963785] Lustre: Skipped 1 previous similar message [ 6837.046688] Lustre: lustre-MDT0001: new disk, initializing [ 6837.183593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6837.248748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6837.273749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6842.172939] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6851.411533] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6857.481826] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6864.725635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6865.054367] Lustre: lustre-OST0000: new disk, initializing [ 6865.057993] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6865.064955] Lustre: 152437:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6865.149681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6870.612243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6870.633931] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6870.793865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6871.393433] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6881.811190] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6888.456659] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_module obdecho/obdecho [ 6890.472049] LustreError: 153440:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6890.481577] LustreError: 153440:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6892.422592] LustreError: 153537:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6903.776474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6903.779822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6903.797942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6906.849706] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6906.863089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6906.875594] Lustre: Skipped 2 previous similar messages [ 6909.479403] Lustre: server umount lustre-MDT0000 complete [ 6913.390294] LustreError: 150461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634991 with bad export cookie 510170820144270319 [ 6913.396616] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6913.405911] LustreError: 150461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6913.777415] Lustre: server umount lustre-MDT0001 complete [ 6928.318781] Lustre: server umount lustre-OST0000 complete [ 6950.948592] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 6962.808572] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7008.280742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7008.552996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7008.599948] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7008.753521] Lustre: lustre-MDT0000: new disk, initializing [ 7008.882948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7008.916873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7013.977427] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7025.281263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7025.403252] Lustre: 157259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7025.453395] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7025.457359] Lustre: Skipped 1 previous similar message [ 7025.500265] Lustre: lustre-MDT0001: new disk, initializing [ 7025.551586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7025.578741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7025.584876] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7032.465528] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7043.571888] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7050.442794] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7058.375708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7058.658445] Lustre: lustre-OST0000: new disk, initializing [ 7058.661346] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7058.669773] Lustre: 159226:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7058.762410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7065.182802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7065.195393] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7065.326428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7066.666760] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7076.548164] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7082.087725] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7082.385923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7088.564722] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7088.776857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7091.174152] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7091.175601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7091.181525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7091.221349] Lustre: Skipped 2 previous similar messages [ 7096.302966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7096.309275] Lustre: Skipped 1 previous similar message [ 7097.020497] Lustre: server umount lustre-OST0000 complete [ 7101.415764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7101.428254] Lustre: Skipped 1 previous similar message [ 7101.436943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7101.470425] Lustre: Skipped 2 previous similar messages [ 7107.543295] Lustre: server umount lustre-MDT0000 complete [ 7111.651951] LustreError: 157272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7111.677256] LustreError: 157272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7112.040440] LustreError: 157253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635190 with bad export cookie 510170820144271278 [ 7112.045666] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7112.052135] LustreError: 157253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7112.528610] Lustre: server umount lustre-MDT0001 complete [ 7116.645708] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 11:33:12 (1786635192) [ 7119.059477] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7121.197987] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 11:33:16 (1786635196) [ 7130.076459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7130.660108] LustreError: 161641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7130.774048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7136.240994] LustreError: 161642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7136.425469] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7141.345044] LustreError: 161641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7145.644344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7151.265567] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7161.658629] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7168.846993] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7176.546285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7177.064559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7177.067633] Lustre: Skipped 1 previous similar message [ 7184.333823] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7192.386485] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7210.977337] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7210.981544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7210.998122] Lustre: Skipped 1 previous similar message [ 7211.018784] Lustre: Skipped 4 previous similar messages [ 7215.476769] Lustre: server umount lustre-MDT0000 complete [ 7216.107162] LustreError: 163422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7216.134435] LustreError: 163422:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7219.599031] LustreError: 161623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635298 with bad export cookie 510170820144272160 [ 7219.626821] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7221.218179] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7221.240784] Lustre: Skipped 1 previous similar message [ 7224.230840] LustreError: 164133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7224.260331] LustreError: 164133:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7226.323838] Lustre: server umount lustre-MDT0001 complete [ 7234.905630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7250.811105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7251.135251] LustreError: 165507:0:(llog.c:1655:llog_backup()) MGC192.168.201.105@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7251.147364] Lustre: 165507:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.105@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7254.381954] LustreError: 165512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7289.824611] LustreError: 165532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7289.848753] LustreError: 165532:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [ 7311.841942] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7311.856267] Lustre: 165507:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.105@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7352.839493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7354.662405] LustreError: 165513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7354.677727] LustreError: 165513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 45 previous similar messages [ 7357.938102] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7357.940544] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7367.631030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7381.484467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7390.712350] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7147da86d782c79 [ 7390.733200] Lustre: MGC192.168.201.105@tcp: Connection restored to 0@lo (at 0@lo) [ 7395.944126] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7396.349282] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7404.151620] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7406.318294] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7412.714409] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7414.432665] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7432.167140] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7432.177880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7432.190491] Lustre: Skipped 1 previous similar message [ 7432.191642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7432.196928] Lustre: Skipped 6 previous similar messages [ 7437.839484] Lustre: server umount lustre-MDT0000 complete [ 7441.949253] LustreError: 161621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635520 with bad export cookie 510170820144278649 [ 7441.974097] LustreError: 161621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7441.979972] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7442.274341] Lustre: server umount lustre-MDT0001 complete [ 7455.858391] Lustre: server umount lustre-OST0000 complete [ 7478.736262] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 7491.578173] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7541.661741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7541.953668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7541.991212] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7542.127572] Lustre: lustre-MDT0000: new disk, initializing [ 7542.212656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7542.219309] Lustre: Skipped 1 previous similar message [ 7542.239255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7548.789306] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7560.759049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7560.911606] Lustre: 170620:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7561.159897] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7565.807301] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7577.078813] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7584.741278] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7592.829741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7593.102357] Lustre: lustre-OST0000: new disk, initializing [ 7593.105557] Lustre: Skipped 1 previous similar message [ 7593.108710] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7593.116479] Lustre: Skipped 2 previous similar messages [ 7593.125737] Lustre: 172586:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7600.196815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7600.216087] Lustre: Skipped 1 previous similar message [ 7600.235193] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7600.399678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7601.690363] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7612.579542] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7618.057344] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7618.271984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7623.817854] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7624.110662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7631.335499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7631.338981] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7631.367623] Lustre: Skipped 1 previous similar message [ 7631.379511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7631.380799] LustreError: Skipped 1 previous similar message [ 7631.414374] Lustre: Skipped 4 previous similar messages [ 7633.005504] Lustre: server umount lustre-OST0000 complete [ 7643.668966] Lustre: server umount lustre-MDT0000 complete [ 7646.688801] LustreError: 170627:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7646.730919] LustreError: 170627:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 32 previous similar messages [ 7648.705297] LustreError: 171423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635727 with bad export cookie 510170820144295456 [ 7648.711752] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7648.714104] LustreError: 171423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7649.205824] Lustre: server umount lustre-MDT0001 complete [ 7659.386873] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:42:14 (1786635734) [ 7667.904923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7674.465898] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7683.801572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7688.765199] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7695.971362] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7701.721549] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7708.688402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7716.588744] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7725.973038] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7739.826874] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:43:35 (1786635815) [ 7744.993473] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7745.002734] LustreError: Skipped 1 previous similar message [ 7745.007321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7745.015309] Lustre: Skipped 3 previous similar messages [ 7745.027384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7745.035074] Lustre: Skipped 3 previous similar messages [ 7750.476560] Lustre: server umount lustre-OST0000 complete [ 7761.699531] Lustre: server umount lustre-MDT0000 complete [ 7765.430726] LustreError: 174982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786635844 with bad export cookie 510170820144296331 [ 7765.432607] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7765.452367] LustreError: 174982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7765.924545] Lustre: server umount lustre-MDT0001 complete [ 7779.042512] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7782.142682] Key type lgssc unregistered [ 7782.402938] LNet: 178734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7782.411629] LNetError: 178734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7782.425871] LNet: Removed LNI 192.168.201.105@tcp [ 7783.317899] Key type .llcrypt unregistered [ 7783.319636] Key type ._llcrypt unregistered [ 7802.222600] Key type ._llcrypt registered [ 7802.226362] Key type .llcrypt registered [ 7802.370669] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7804.020020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7804.058239] alg: No test for adler32 (adler32-zlib) [ 7805.247894] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 7805.577826] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 7807.344634] Key type lgssc registered [ 7808.616484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7817.241021] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7817.261702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7832.799518] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7837.920244] LustreError: 179979:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.105@tcp: failed processing log, type 4: rc = -110 [ 7843.040599] LustreError: 179979:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7843.069178] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7843.080127] LustreError: MGC192.168.201.105@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7843.106248] LustreError: 179979:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7843.127774] LustreError: 179979:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7843.148378] LustreError: 179979:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7843.230521] Lustre: server umount lustre-OST0000 complete [ 7843.233806] LustreError: 179979:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7851.716815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7853.130472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7856.824766] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7863.281372] LustreError: 180479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7863.299347] LustreError: 180479:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7865.843650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7866.368720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7870.896670] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7878.917490] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7885.462293] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7892.552168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7892.748200] LustreError: 182294:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7892.760181] Lustre: 182294:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.105@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7892.876687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7899.232452] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7908.879868] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7912.417952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7912.428229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7912.477202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7916.528162] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7916.536225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7916.541792] Lustre: Skipped 1 previous similar message [ 7916.566197] Lustre: Skipped 2 previous similar messages [ 7918.266110] Lustre: server umount lustre-MDT0000 complete [ 7921.634721] LustreError: 180479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7921.646516] LustreError: 180479:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7922.170410] LustreError: 180459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786636000 with bad export cookie 13322921157297376988 [ 7922.177857] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7922.180718] LustreError: 180459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7922.601972] Lustre: server umount lustre-MDT0001 complete [ 7937.453421] Lustre: server umount lustre-OST0000 complete [ 7948.997471] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 7965.028270] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8006.119444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8006.364175] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8006.398160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8006.475233] Lustre: lustre-MDT0000: new disk, initializing [ 8006.553097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8006.563958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8012.336986] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8027.886808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8027.938732] Lustre: 185844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8027.960420] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8027.962098] Lustre: Skipped 1 previous similar message [ 8027.995176] Lustre: lustre-MDT0001: new disk, initializing [ 8028.086580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8028.111549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8028.135942] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8032.721670] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8042.490224] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8048.061593] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8054.413953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8054.649783] Lustre: lustre-OST0000: new disk, initializing [ 8054.653804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8054.667186] Lustre: 187815:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8054.751595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8061.175762] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8061.487957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8061.498052] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8061.602987] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8071.684644] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8077.294228] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8077.512578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8081.946643] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8082.150416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8087.013386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8087.027789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8087.046472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8090.889658] Lustre: server umount lustre-OST0000 complete [ 8097.764707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8097.777041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8097.796294] Lustre: Skipped 2 previous similar messages [ 8097.804349] Lustre: Skipped 1 previous similar message [ 8102.118410] Lustre: server umount lustre-MDT0000 complete [ 8102.883662] LustreError: 187007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8102.917890] LustreError: 187007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8105.848646] LustreError: 187814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786636184 with bad export cookie 13322921157297377709 [ 8105.850079] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8105.862773] LustreError: 187814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8106.147212] Lustre: server umount lustre-MDT0001 complete [ 8114.371313] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:49:50 (1786636190) [ 8132.237691] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8134.748673] Key type lgssc unregistered [ 8135.022930] LNet: 190761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8135.030938] LNetError: 190761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8135.047099] LNet: Removed LNI 192.168.201.105@tcp [ 8136.226923] Key type .llcrypt unregistered [ 8136.230597] Key type ._llcrypt unregistered [ 8153.753805] Key type ._llcrypt registered [ 8153.755863] Key type .llcrypt registered [ 8153.830082] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8154.635839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8154.666786] alg: No test for adler32 (adler32-zlib) [ 8155.786275] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 8156.037506] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 8157.679848] Key type lgssc registered [ 8158.880865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8167.065568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8167.078603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8168.670690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8173.505713] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8179.175245] LustreError: 191980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8179.197077] LustreError: 191980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8183.886747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8184.478299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8188.808111] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8198.023034] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8204.655489] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8212.250874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8212.447345] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8212.451636] LustreError: 193846:0:(llog.c:1696:llog_backup()) MGC192.168.201.105@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8212.458846] Lustre: 193846:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.105@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8212.638305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8219.385224] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8227.754415] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8230.371274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8230.381564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8230.408167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8234.471494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8234.474873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8234.481994] Lustre: Skipped 1 previous similar message [ 8234.498500] Lustre: Skipped 1 previous similar message [ 8236.435771] Lustre: server umount lustre-MDT0000 complete [ 8239.596132] LustreError: 193930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8239.612627] LustreError: 193930:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8239.823615] LustreError: 191961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786636318 with bad export cookie 17419005805403041310 [ 8239.833545] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8239.836186] LustreError: 191961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8240.321264] Lustre: server umount lustre-MDT0001 complete [ 8254.546578] Lustre: server umount lustre-OST0000 complete [ 8267.075322] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 8279.558619] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8317.358308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8317.650399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8317.690673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8317.811732] Lustre: lustre-MDT0000: new disk, initializing [ 8317.874965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8317.907821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8323.433618] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8336.169635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8336.309521] Lustre: 197370:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8336.366717] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8336.375839] Lustre: Skipped 1 previous similar message [ 8336.382687] LustreError: 198155:0:(llog.c:1696:llog_backup()) MGC192.168.201.105@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8336.396902] LustreError: 198155:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8336.409741] Lustre: 198155:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.105@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8336.470968] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8336.477430] Lustre: Skipped 1 previous similar message [ 8336.603092] Lustre: lustre-MDT0001: new disk, initializing [ 8336.781618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8336.845689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8336.862832] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8342.081746] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8353.827700] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8361.222410] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8370.573414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8371.123761] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8371.127834] Lustre: Skipped 1 previous similar message [ 8371.138665] LustreError: 199316:0:(llog.c:1696:llog_backup()) MGC192.168.201.105@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8371.154960] LustreError: 199316:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8371.167085] Lustre: 199316:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.105@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8371.191043] Lustre: 199316:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8371.223951] Lustre: lustre-OST0000: new disk, initializing [ 8371.228576] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8371.240595] Lustre: 199340:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8371.334837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8377.441384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8377.449172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8377.624881] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8378.668913] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8389.829273] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8395.532344] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8395.805591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8401.342507] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8401.688970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8408.549534] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8408.564517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8408.567144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8408.579117] Lustre: Skipped 1 previous similar message [ 8408.589978] Lustre: Skipped 1 previous similar message [ 8413.666016] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8413.680154] Lustre: Skipped 2 previous similar messages [ 8418.271990] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8418.801074] Lustre: server umount lustre-OST0000 complete [ 8423.906404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8423.908283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8423.914291] LustreError: Skipped 1 previous similar message [ 8423.943246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8423.956361] Lustre: Skipped 1 previous similar message [ 8434.148420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8434.154969] Lustre: Skipped 2 previous similar messages [ 8437.216680] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8437.811625] Lustre: server umount lustre-MDT0000 complete [ 8439.264383] LustreError: 199834:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8439.283256] LustreError: 199834:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8441.687326] LustreError: 197362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786636520 with bad export cookie 17419005805403042031 [ 8441.693395] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8441.699375] LustreError: 197362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8441.957397] Lustre: server umount lustre-MDT0001 complete [ 8450.231331] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:55:25 (1786636525) [ 8467.876990] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8476.951876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8477.192836] LustreError: 202473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8477.229963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8480.689336] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8482.273333] LustreError: 202473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8488.049929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8492.538357] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8495.166694] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8501.931687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8508.715884] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8517.251240] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8517.487386] Lustre: lustre-OST0001: new disk, initializing [ 8517.493619] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8517.500331] Lustre: 204783:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8517.629136] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8517.635106] Lustre: Skipped 2 previous similar messages [ 8523.353672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8523.364612] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8523.482165] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8523.769512] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8534.244619] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8540.702970] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8542.527557] Lustre: DEBUG MARKER: ADD OST3 [ 8547.188026] loop: module loaded [ 8547.254512] loop0: detected capacity change from 0 to 204800000 [ 8547.481379] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8547.490128] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8547.499928] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8553.843335] loop0: detected capacity change from 0 to 204800000 [ 8554.963886] Lustre: DEBUG MARKER: STOP OST3 [ 8560.106965] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8560.323546] Lustre: lustre-OST0003: new disk, initializing [ 8560.328729] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8560.340552] Lustre: 206779:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8562.017915] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8562.123391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8562.128685] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8562.186386] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8564.063078] LustreError: 206954:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8564.299101] Lustre: Failing over lustre-OST0003 [ 8564.622654] Lustre: server umount lustre-OST0003 complete [ 8566.594954] LustreError: 204512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8566.614246] LustreError: 204512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8567.289933] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8567.296249] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8567.305294] Lustre: Skipped 1 previous similar message [ 8567.580793] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8576.450916] Lustre: DEBUG MARKER: START OST3 again [ 8582.641120] LustreError: 203959:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8582.656166] LustreError: 203959:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 8583.070139] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8583.452512] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8583.460933] Lustre: Skipped 1 previous similar message [ 8583.484595] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8585.065149] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8585.384868] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8585.386552] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8590.752473] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8613.186083] Lustre: Failing over lustre-OST0003 [ 8613.425910] Lustre: server umount lustre-OST0003 complete [ 8614.397827] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8614.420984] Lustre: Skipped 1 previous similar message [ 8619.511682] LustreError: 203959:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8619.525708] LustreError: 203959:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8634.848669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8634.862235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8634.874247] Lustre: Skipped 3 previous similar messages [ 8634.879780] Lustre: Skipped 3 previous similar messages [ 8638.395478] Lustre: server umount lustre-MDT0000 complete [ 8642.320055] LustreError: 202454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786636720 with bad export cookie 17419005805403042913 [ 8642.332367] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8642.345449] LustreError: 202454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8648.799947] Lustre: server umount lustre-MDT0001 complete [ 8652.784304] Lustre: server umount lustre-OST0000 complete [ 8656.951487] Lustre: server umount lustre-OST0001 complete [ 8664.128219] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 8672.120267] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8708.752421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8708.940985] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8708.960390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8709.024963] Lustre: lustre-MDT0000: new disk, initializing [ 8709.100210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8714.650654] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8726.632537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8726.819412] Lustre: 212240:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8726.932742] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8726.937723] Lustre: Skipped 1 previous similar message [ 8727.056278] Lustre: lustre-MDT0001: new disk, initializing [ 8727.212453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8727.223941] Lustre: Skipped 1 previous similar message [ 8727.257626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8727.280717] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8732.894529] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8746.175632] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8753.167609] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8760.729759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8761.076920] Lustre: lustre-OST0000: new disk, initializing [ 8761.083728] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8761.098310] Lustre: 214214:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8762.915907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8762.937496] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8763.101915] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8768.268738] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8778.200966] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8783.993948] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8784.408411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8790.269779] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8790.548369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8794.082983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8794.088858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8794.110606] Lustre: Skipped 3 previous similar messages [ 8794.132777] Lustre: Skipped 5 previous similar messages [ 8798.915690] Lustre: server umount lustre-OST0000 complete [ 8804.321083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8809.735667] Lustre: server umount lustre-MDT0000 complete [ 8813.539857] LustreError: 212253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8813.564962] LustreError: 212253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 8814.494400] LustreError: 212234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786636893 with bad export cookie 17419005805403045209 [ 8814.496552] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8814.520351] LustreError: 212234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8815.284635] Lustre: server umount lustre-MDT0001 complete [ 8819.590710] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 12:01:35 (1786636895) [ 8837.083896] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 8850.012458] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8888.338868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8888.725583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8888.814935] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8888.906811] Lustre: lustre-MDT0000: new disk, initializing [ 8889.060970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8894.723167] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8905.404377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8905.553098] Lustre: 218841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8905.933273] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8910.842819] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8920.135602] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8925.060938] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8931.492406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8931.749181] Lustre: 220807:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8933.982076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8937.042297] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8947.330400] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8952.082353] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8952.244585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8956.204358] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8956.408627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8959.458597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8959.471372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8959.477278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8959.486613] Lustre: Skipped 3 previous similar messages [ 8959.498798] Lustre: Skipped 5 previous similar messages [ 8964.309594] Lustre: server umount lustre-OST0000 complete [ 8974.244384] Lustre: server umount lustre-MDT0000 complete [ 8974.817545] LustreError: 218852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8974.836215] LustreError: 218852:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8977.858577] LustreError: 218833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637056 with bad export cookie 17419005805403046091 [ 8977.863987] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8977.870319] LustreError: 218833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8978.398613] Lustre: server umount lustre-MDT0001 complete [ 8987.670937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8988.510958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8988.518085] Lustre: Skipped 4 previous similar messages [ 8994.244625] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9004.801882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9009.949652] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9020.171130] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9027.183294] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9034.605750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9042.863402] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9050.493102] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9074.162937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9076.838245] Lustre: server umount lustre-OST0000 complete [ 9089.505532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9089.512969] Lustre: Skipped 8 previous similar messages [ 9094.111142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9094.258216] Lustre: server umount lustre-MDT0000 complete [ 9098.132037] LustreError: 223005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637176 with bad export cookie 17419005805403046973 [ 9098.149542] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9098.153338] LustreError: 223005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9098.507416] Lustre: server umount lustre-MDT0001 complete [ 9190.422184] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 9193.048967] Key type lgssc unregistered [ 9193.295299] LNet: 226711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9193.304360] LNetError: 226711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9193.322704] LNet: Removed LNI 192.168.201.105@tcp [ 9194.138158] Key type .llcrypt unregistered [ 9194.139861] Key type ._llcrypt unregistered [ 9220.729908] Key type ._llcrypt registered [ 9220.733941] Key type .llcrypt registered [ 9220.827910] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 9237.898514] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 9239.491551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9239.512769] alg: No test for adler32 (adler32-zlib) [ 9240.560176] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [ 9240.782619] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 9242.447168] Key type lgssc registered [ 9243.580043] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9275.014290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9275.036989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9276.269185] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9276.294855] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9276.371267] Lustre: lustre-MDT0000: new disk, initializing [ 9276.434291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9276.449714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9279.999810] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9289.533727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9289.696575] Lustre: 230064:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9289.780557] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9289.784223] Lustre: Skipped 1 previous similar message [ 9289.866812] Lustre: lustre-MDT0001: new disk, initializing [ 9289.939798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9289.973697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9289.990783] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9293.978568] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9303.533948] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9309.718935] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9316.524289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9316.805889] Lustre: lustre-OST0000: new disk, initializing [ 9316.812652] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9316.819395] Lustre: 232033:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9316.911499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9324.295830] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9327.161874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9327.174119] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9327.277481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9334.068804] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9338.477184] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9338.678429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9343.243926] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9343.413297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9348.077042] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9348.078635] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9348.087488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9348.091960] Lustre: Skipped 1 previous similar message [ 9348.122405] LustreError: Skipped 1 previous similar message [ 9352.006481] Lustre: server umount lustre-OST0000 complete [ 9356.782656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9356.792072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9356.805253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9356.807356] Lustre: Skipped 1 previous similar message [ 9358.307418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9358.308067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9358.349407] Lustre: Skipped 1 previous similar message [ 9363.424638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9363.432762] Lustre: Skipped 1 previous similar message [ 9367.523464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9367.532858] Lustre: Skipped 1 previous similar message [ 9370.079140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9370.290808] Lustre: server umount lustre-MDT0000 complete [ 9372.646474] LustreError: 231398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9372.670603] LustreError: 231398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9373.643760] LustreError: 230873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637452 with bad export cookie 5438086834345517719 [ 9373.645830] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9373.658705] LustreError: 230873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9373.913876] Lustre: server umount lustre-MDT0001 complete [ 9391.324121] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 9400.287173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9400.787658] LustreError: 234968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9400.850196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9405.566118] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9405.921865] LustreError: 234969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9411.049400] LustreError: 234968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9416.167811] LustreError: 234969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9416.881931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9417.428588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9422.302649] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9425.438346] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9433.458777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9433.970628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9440.407143] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9449.660704] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9449.943378] Lustre: lustre-OST0001: new disk, initializing [ 9449.947089] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9449.964647] Lustre: 237274:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9450.154376] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9450.706381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9450.721738] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9450.898347] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9456.892700] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9469.796378] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9480.834488] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9493.841603] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 12:12:49 (1786637569) [ 9497.065201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9497.075051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9497.081195] Lustre: Skipped 1 previous similar message [ 9497.106285] Lustre: Skipped 3 previous similar messages [ 9510.367159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9510.696925] Lustre: server umount lustre-MDT0000 complete [ 9512.441633] LustreError: 234968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9512.464739] LustreError: 234968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9514.459817] LustreError: 237273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637593 with bad export cookie 5438086834345518601 [ 9514.467384] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9514.472819] LustreError: 237273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9514.927861] Lustre: server umount lustre-MDT0001 complete [ 9528.729794] Lustre: server umount lustre-OST0000 complete [ 9543.400212] Lustre: server umount lustre-OST0001 complete [ 9550.392459] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 9561.154240] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 9597.015676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9597.304856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9597.325411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9597.382413] Lustre: lustre-MDT0000: new disk, initializing [ 9597.423107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9597.432651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9602.384576] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9614.763146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9614.887686] Lustre: 241227:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9614.927896] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9614.931115] Lustre: Skipped 1 previous similar message [ 9614.988703] Lustre: lustre-MDT0001: new disk, initializing [ 9615.121902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9615.137979] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9619.607953] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9628.173341] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9633.380576] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9640.633286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9640.859818] Lustre: lustre-OST0000: new disk, initializing [ 9640.864046] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9640.873263] Lustre: 243195:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9642.562275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9642.576686] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9642.739855] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9646.544413] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9655.605331] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9661.391324] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9661.624276] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9666.641480] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9666.833215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9668.583143] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9668.591593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9668.596290] LustreError: Skipped 1 previous similar message [ 9668.599723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9668.599732] Lustre: Skipped 8 previous similar messages [ 9668.647250] Lustre: Skipped 3 previous similar messages [ 9674.727021] Lustre: server umount lustre-OST0000 complete [ 9678.817652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9678.843073] Lustre: Skipped 1 previous similar message [ 9692.127214] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9692.523502] Lustre: server umount lustre-MDT0000 complete [ 9694.183646] LustreError: 241237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9694.202463] LustreError: 241237:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9695.750110] LustreError: 241219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786637774 with bad export cookie 5438086834345520736 [ 9695.758060] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9695.764069] LustreError: 241219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9696.101482] Lustre: server umount lustre-MDT0001 complete [ 9703.959843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9704.500729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9704.507290] Lustre: Skipped 2 previous similar messages [ 9709.000485] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9717.834613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9722.800426] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9729.935661] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9734.812553] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9741.581374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9748.474516] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9756.182971] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9907.169733] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9907.175283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9907.184942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9907.194880] Lustre: Skipped 9 previous similar messages [ 9910.492352] Lustre: server umount lustre-OST0000 complete [ 9917.922524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9917.939690] LustreError: Skipped 1 previous similar message [ 9921.014071] Lustre: server umount lustre-MDT0000 complete [ 9924.578939] LustreError: 245413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9924.599105] LustreError: 245413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9924.783882] LustreError: 245396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786638003 with bad export cookie 5438086834345521611 [ 9924.788525] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9924.791952] LustreError: 245396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9925.202528] Lustre: server umount lustre-MDT0001 complete [10026.003947] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [10030.057236] Key type lgssc unregistered [10030.333744] LNet: 249025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10030.338804] LNetError: 249025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10030.354447] LNet: Removed LNI 192.168.201.105@tcp [10031.470438] Key type .llcrypt unregistered [10031.482445] Key type ._llcrypt unregistered [10065.291987] Key type ._llcrypt registered [10065.296949] Key type .llcrypt registered [10065.506538] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [10082.651315] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10084.301120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10084.371953] alg: No test for adler32 (adler32-zlib) [10085.657243] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [10085.887186] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [10087.559188] Key type lgssc registered [10088.778717] Lustre: Echo OBD driver; http://www.lustre.org/ [10147.330328] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10161.255218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10161.294242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10162.611574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10162.664222] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10162.786822] Lustre: lustre-MDT0000: new disk, initializing [10162.886444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10162.907901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10168.198847] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10181.173584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10181.377149] Lustre: 253119:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10181.542243] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10181.559982] Lustre: Skipped 1 previous similar message [10181.685463] Lustre: lustre-MDT0001: new disk, initializing [10181.803237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10181.857569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10181.884922] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10186.696409] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10191.998328] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10198.894843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10199.141649] Lustre: lustre-OST0000: new disk, initializing [10199.144750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10199.151106] Lustre: 254751:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10199.238962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10201.552097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10201.561607] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10201.775225] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10205.060499] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10218.363316] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10218.545575] Lustre: lustre-OST0001: new disk, initializing [10218.548610] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10218.553940] Lustre: 255626:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10218.666518] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10225.294185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10225.313264] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10225.476370] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10226.612295] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10238.626420] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10250.579932] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10264.032550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10264.050137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10264.070212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10266.596026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10266.597866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10266.627725] Lustre: Skipped 1 previous similar message [10266.638683] Lustre: Skipped 3 previous similar messages [10269.068704] Lustre: server umount lustre-MDT0000 complete [10274.089650] LustreError: 253112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786638352 with bad export cookie 10139659561226899854 [10274.106180] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10274.123145] LustreError: 253112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10274.688175] Lustre: server umount lustre-MDT0001 complete [10290.229595] Lustre: server umount lustre-OST0000 complete [10293.218933] Lustre: 250603:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786638355/real 1786638355] req@ffff912c5159f100 x1873425899538432/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786638371 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10293.243885] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10293.259868] Lustre: Skipped 1 previous similar message [10294.333834] Lustre: server umount lustre-OST0001 complete [10308.973996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10319.212582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10330.972143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10343.054987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10368.438697] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10379.800339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10379.851641] Lustre: lustre-MDT0000: reset Object Index mappings [10380.279113] LustreError: 260637:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10380.288461] LustreError: 260637:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10380.350040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10384.936167] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10385.377838] LustreError: 260638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10389.475600] LustreError: 260637:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10394.101301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10394.117597] Lustre: lustre-MDT0001: reset Object Index mappings [10394.509377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10399.491235] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10402.553024] Lustre: 261780:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10410.812447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10411.279776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10417.760741] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10419.501119] LustreError: 262137:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10424.628240] LustreError: 262137:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10424.657754] LustreError: 262137:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10424.677214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10427.469419] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10432.641481] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10433.031586] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10439.100422] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10442.637787] Lustre: 263652:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10448.355752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10448.359251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10448.370850] Lustre: Skipped 2 previous similar messages [10448.382418] Lustre: Skipped 3 previous similar messages [10449.315968] Lustre: server umount lustre-MDT0000 complete [10452.566970] LustreError: 261782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786638531 with bad export cookie 10139659561226906644 [10452.568083] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10452.573144] LustreError: 261782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10452.975533] Lustre: server umount lustre-MDT0001 complete [10466.717982] Lustre: server umount lustre-OST0000 complete [10469.858386] Lustre: 250603:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786638532/real 1786638532] req@ffff912d77b29180 x1873425899586432/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786638548 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10469.875836] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10469.887317] Lustre: Skipped 1 previous similar message [10470.904266] Lustre: server umount lustre-OST0001 complete [10488.274592] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 12:29:24 (1786638564) [10504.629573] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [10513.122713] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10545.237761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10545.494748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10545.515808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10545.614566] Lustre: lustre-MDT0000: new disk, initializing [10545.721305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10545.723742] Lustre: Skipped 1 previous similar message [10545.734201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10550.236483] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10561.331449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10561.423180] Lustre: 267920:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10561.462634] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10561.465220] Lustre: Skipped 1 previous similar message [10561.516187] Lustre: lustre-MDT0001: new disk, initializing [10561.617239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10561.626499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10565.582863] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10575.495772] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10580.903204] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10587.716335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10588.003873] Lustre: lustre-OST0000: new disk, initializing [10588.011352] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10588.018723] Lustre: 269887:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10589.340356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10589.355329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10589.423736] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10593.703337] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10603.737877] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10607.682619] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10607.852796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10611.765840] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10611.985234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10615.268797] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10615.276160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10615.291449] Lustre: Skipped 1 previous similar message [10615.301030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10615.306808] Lustre: Skipped 1 previous similar message [10620.384674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10628.063221] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10628.288621] Lustre: server umount lustre-OST0000 complete [10633.186093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10633.196919] LustreError: Skipped 1 previous similar message [10633.200480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10633.213633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10633.216085] Lustre: Skipped 3 previous similar messages [10637.890551] Lustre: server umount lustre-MDT0000 complete [10640.865287] LustreError: 267926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10640.885840] LustreError: 267926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10641.190356] LustreError: 269886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786638719 with bad export cookie 10139659561226908730 [10641.198061] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10641.210468] LustreError: 269886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10641.527892] Lustre: server umount lustre-MDT0001 complete [10656.625034] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10665.116610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10665.474920] LustreError: 272822:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10665.524715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10665.530169] Lustre: Skipped 2 previous similar messages [10669.242937] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10677.517387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10682.283171] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10685.161772] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10691.721986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10698.089216] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10707.359841] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10707.564082] Lustre: lustre-OST0001: new disk, initializing [10707.567197] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10707.572549] Lustre: 275128:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10709.061360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10709.077651] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10709.180592] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10713.561371] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10722.686061] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10725.621459] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10730.188949] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10730.494816] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10735.926439] LustreError: 250605:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff912d49108e00 x1873425899683456/t0(0) o6->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 [10736.212557] Lustre: server umount lustre-MDT0000 complete [10737.130957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10737.148875] Lustre: Skipped 2 previous similar messages [10737.159241] LustreError: 275705:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10737.183163] LustreError: 275705:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10739.037525] LustreError: 272803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786638817 with bad export cookie 10139659561226909612 [10739.039023] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10739.047296] LustreError: 272803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10741.439406] Lustre: server umount lustre-MDT0001 complete [10754.533026] Lustre: server umount lustre-OST0000 complete [10768.441030] Lustre: server umount lustre-OST0001 complete [10785.330377] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10795.612550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10796.179587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10796.187509] Lustre: Skipped 3 previous similar messages [10801.647410] LustreError: 278183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10801.660344] LustreError: 278183:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10801.792935] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10813.047409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10819.176593] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10822.290740] Lustre: 279324:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10829.195045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10835.161673] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10840.906257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10845.250506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10850.821205] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10851.971556] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10858.710794] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10862.209852] Lustre: 281200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10866.375517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10866.400283] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10874.497682] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 12:35:50 (1786638950) [10890.211408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10890.220740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10890.236163] Lustre: Skipped 2 previous similar messages [10890.247826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10890.260368] Lustre: Skipped 2 previous similar messages [10894.647618] Lustre: server umount lustre-MDT0000 complete [10898.344265] LustreError: 278164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786638976 with bad export cookie 10139659561226915961 [10898.350241] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10898.354122] LustreError: 278164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10904.876537] Lustre: server umount lustre-MDT0001 complete [10914.350978] Lustre: server umount lustre-OST0000 complete [10925.097770] Lustre: server umount lustre-OST0001 complete [10932.100409] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [10941.993842] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10985.520672] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10995.497098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10995.704831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10995.728580] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10995.804758] Lustre: lustre-MDT0000: new disk, initializing [10995.888609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10999.836337] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11008.796903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11008.907700] Lustre: 285440:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11009.105215] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11013.211604] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11023.531337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11023.901897] Lustre: 287072:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11025.823169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11030.381389] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11040.058134] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11040.187464] Lustre: 287944:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11041.576553] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11041.589787] Lustre: Skipped 1 previous similar message [11041.694135] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11046.338504] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11055.483699] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11059.016569] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11059.021090] Lustre: Skipped 1 previous similar message [11071.910665] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 12:39:07 (1786639147) [11075.555915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11075.563453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11075.579096] Lustre: Skipped 5 previous similar messages [11075.587145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11075.604682] Lustre: Skipped 8 previous similar messages [11080.410835] Lustre: server umount lustre-MDT0000 complete [11082.725115] LustreError: 289030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11082.746841] LustreError: 289030:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [11083.805040] LustreError: 286243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786639162 with bad export cookie 10139659561226923850 [11083.810723] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11083.818761] LustreError: 286243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11084.125557] Lustre: server umount lustre-MDT0001 complete [11098.452711] Lustre: server umount lustre-OST0000 complete [11112.790761] Lustre: server umount lustre-OST0001 complete [11122.103414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11122.700688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11122.712260] Lustre: Skipped 7 previous similar messages [11127.791879] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11136.592484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11143.324368] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11152.467254] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11159.758130] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11167.389646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11174.417518] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11183.050718] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11199.343633] loop1: detected capacity change from 0 to 204800000 [11199.540610] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11199.556345] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11199.572588] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11205.194902] loop1: detected capacity change from 0 to 204800000 [11211.670972] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11211.790658] LustreError: 290180:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs26712: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11211.805954] LustreError: 290180:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs26712 [11211.812155] LustreError: 290180:0:(mgs_handler.c:517:mgs_target_reg()) fs26712-MDT0001: Error checking target index: rc = -13 [11211.830063] LustreError: fs26712-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11211.839073] LustreError: 293909:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11211.855869] LustreError: 293909:0:(tgt_mount.c:2129:server_put_super()) no obd fs26712-MDT0001 [11211.861180] LustreError: 293909:0:(tgt_mount.c:116:server_deregister_mount()) fs26712-MDT0001 not registered [11211.886313] Lustre: server umount fs26712-MDT0001 complete [11211.888853] LustreError: 293909:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11221.860797] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11222.119238] Lustre: Setting parameter fs26712-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs26712-MDT0001 [11222.197606] Lustre: srv-fs26712-MDT0001: No data found on store. Initialize space. [11222.212641] Lustre: Skipped 4 previous similar messages [11222.297294] Lustre: fs26712-MDT0001: new disk, initializing [11222.300740] Lustre: Skipped 3 previous similar messages [11222.490394] Lustre: srv-fs26712-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11222.503758] Lustre: 294376:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs26712-MDT0001-mdtlov: can't create new llog: rc = -115 [11222.521378] LustreError: 294376:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs26712-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11228.736812] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11236.205709] Lustre: server umount fs26712-MDT0001 complete [11248.633033] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11248.907789] loop1: detected capacity change from 0 to 204800000 [11254.812783] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11255.400669] Lustre: 295942:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs26712-MDT0001-mdtlov: can't create new llog: rc = -115 [11255.415442] LustreError: 295942:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs26712-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11260.270715] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11264.778864] Lustre: server umount fs26712-MDT0001 complete [11275.233068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11275.238791] LustreError: Skipped 1 previous similar message [11275.249789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11275.258070] Lustre: Skipped 4 previous similar messages [11281.260706] Lustre: server umount lustre-MDT0000 complete [11290.926875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11291.129636] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11296.606291] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11296.760456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11305.315451] loop1: detected capacity change from 0 to 204800000 [11311.795580] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11312.502366] Lustre: 298388:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs26712-MDT0001-mdtlov: can't create new llog: rc = -115 [11312.514795] LustreError: 298388:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs26712-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11317.171815] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11322.672421] Lustre: server umount fs26712-MDT0001 complete [11342.633909] LustreError: 292020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11342.652429] LustreError: 292020:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 106 previous similar messages [11345.891536] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11345.905244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11345.938909] Lustre: Skipped 6 previous similar messages [11359.199156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11359.389761] Lustre: server umount lustre-OST0000 complete [11369.254687] Lustre: server umount lustre-MDT0000 complete [11373.092588] LustreError: 290173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786639451 with bad export cookie 10139659561226926531 [11373.120992] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11373.449987] Lustre: server umount lustre-MDT0001 complete [11384.493517] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [11387.084501] Key type lgssc unregistered [11387.414198] LNet: 300737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11387.434350] LNetError: 300737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11387.453505] LNet: Removed LNI 192.168.201.105@tcp [11388.416270] Key type .llcrypt unregistered [11388.419706] Key type ._llcrypt unregistered [11414.268729] Key type ._llcrypt registered [11414.270557] Key type .llcrypt registered [11414.341662] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [11415.264318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11415.525826] alg: No test for adler32 (adler32-zlib) [11416.811684] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [11417.091516] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [11418.895213] Key type lgssc registered [11420.456787] Lustre: Echo OBD driver; http://www.lustre.org/ [11429.871154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11429.901786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11431.592710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11436.382782] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11442.148557] LustreError: 302017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11442.155965] LustreError: 302017:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11445.105673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11445.669390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11450.193026] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11457.785899] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11463.735281] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11470.658681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11471.271224] LustreError: 303845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11471.301503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11476.471319] LustreError: 303844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11479.446731] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11482.600618] LustreError: 304070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11482.635362] LustreError: 304070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11487.719234] LustreError: 304070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11487.740442] LustreError: 304070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11488.133776] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11496.241102] LustreError: 303844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11496.268730] LustreError: 303844:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11507.143632] loop1: detected capacity change from 0 to 204800000 [11507.295731] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11507.307229] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11507.336963] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11512.045504] loop1: detected capacity change from 0 to 204800000 [11513.318727] LustreError: 303845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11513.334709] LustreError: 303845:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [11517.421718] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11517.579520] Lustre: fs13911-OST0000: new disk, initializing [11517.588631] Lustre: srv-fs13911-OST0000: No data found on store. Initialize space. [11517.603830] Lustre: 305750:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11517.679721] Lustre: fs13911-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11523.886961] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11533.176902] Lustre: server umount fs13911-OST0000 complete [11544.468932] loop1: detected capacity change from 0 to 204800000 [11547.443311] LustreError: 303847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11547.459033] LustreError: 303847:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [11549.615704] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11549.852885] Lustre: fs13911-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11554.286076] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11555.998246] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11559.406952] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11563.847156] loop2: detected capacity change from 0 to 204800000 [11564.145448] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11564.163230] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11564.198023] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11569.726153] loop2: detected capacity change from 0 to 204800000 [11575.547215] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11575.619380] LustreError: fs13911-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11575.634687] LustreError: fs13911-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11575.644896] LustreError: 308789:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11575.651883] LustreError: 308789:0:(tgt_mount.c:2129:server_put_super()) no obd fs13911-OST0001 [11575.658418] LustreError: 308789:0:(tgt_mount.c:116:server_deregister_mount()) fs13911-OST0001 not registered [11575.689601] Lustre: server umount fs13911-OST0001 complete [11575.694412] LustreError: 308789:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11586.332698] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11586.434394] Lustre: fs13911-OST0001: new disk, initializing [11586.437974] Lustre: srv-fs13911-OST0001: No data found on store. Initialize space. [11586.443920] Lustre: 309342:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11586.515413] Lustre: fs13911-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11592.630805] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11602.104404] Lustre: server umount fs13911-OST0000 complete [11613.086364] Lustre: server umount fs13911-OST0001 complete [11613.989985] LustreError: 303845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11614.014765] LustreError: 303845:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [11628.668451] loop1: detected capacity change from 0 to 204800000 [11636.026658] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11636.387715] Lustre: fs13911-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11644.034307] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11652.589707] loop2: detected capacity change from 0 to 204800000 [11659.741686] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11666.485526] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11676.009429] Lustre: server umount fs13911-OST0000 complete [11685.168519] Lustre: server umount fs13911-OST0001 complete [11697.632631] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11697.643137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11697.649237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11697.684145] Lustre: Skipped 1 previous similar message [11702.759024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11702.762819] Lustre: Skipped 1 previous similar message [11703.764897] Lustre: server umount lustre-OST0000 complete [11707.883115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11707.904650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11707.912793] Lustre: Skipped 1 previous similar message [11713.012268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11713.018028] Lustre: Skipped 1 previous similar message [11714.287471] Lustre: server umount lustre-MDT0000 complete [11718.252526] LustreError: 301998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786639796 with bad export cookie 12709134898175488249 [11718.268652] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11718.627987] Lustre: server umount lustre-MDT0001 complete [11729.240905] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [11732.040378] Key type lgssc unregistered [11732.291192] LNet: 315269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11732.296962] LNetError: 315269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11732.315279] LNet: Removed LNI 192.168.201.105@tcp [11733.142782] Key type .llcrypt unregistered [11733.144736] Key type ._llcrypt unregistered [11769.180243] Key type ._llcrypt registered [11769.185703] Key type .llcrypt registered [11769.289787] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [11785.343052] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [11786.271924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11786.428500] alg: No test for adler32 (adler32-zlib) [11787.465859] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [11787.750329] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [11789.432058] Key type lgssc registered [11790.282390] Lustre: Echo OBD driver; http://www.lustre.org/ [11824.837544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11824.876737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11826.051481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11826.086987] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11826.190991] Lustre: lustre-MDT0000: new disk, initializing [11826.261643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11826.302864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11831.150494] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11842.761896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11842.981446] Lustre: 319129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11843.084746] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11843.091030] Lustre: Skipped 1 previous similar message [11843.198673] Lustre: lustre-MDT0001: new disk, initializing [11843.347483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11843.410564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11843.428748] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11848.086763] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11858.612640] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11865.079182] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11871.808187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11872.070332] Lustre: lustre-OST0000: new disk, initializing [11872.075383] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11872.087894] Lustre: 321098:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11872.204138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11878.842131] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11879.526311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11879.544512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11879.655379] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11887.504653] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11889.523901] Lustre: Mounted lustre-client - version 2.17.57_2_g7a6d6aa [11890.475612] Lustre: Unmounted lustre-client [11899.876882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11899.899204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11899.918385] Lustre: Skipped 1 previous similar message [11899.929253] Lustre: Skipped 1 previous similar message [11904.992832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11905.003216] Lustre: Skipped 1 previous similar message [11910.116712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11910.131799] Lustre: Skipped 1 previous similar message [11914.209177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11914.496893] Lustre: server umount lustre-OST0000 complete [11920.356475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11920.360646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11920.361055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11920.379356] Lustre: Skipped 1 previous similar message [11924.994341] Lustre: server umount lustre-MDT0000 complete [11928.539068] LustreError: 319124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786640007 with bad export cookie 1567948253447793789 [11928.558456] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11928.566420] LustreError: 319124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11928.881455] Lustre: server umount lustre-MDT0001 complete [11940.656851] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [11943.456201] LNet: 1 local NIs in recovery (showing 1): 192.168.201.105@tcp [11944.031255] Key type lgssc unregistered [11944.281775] LNet: 323263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11944.290660] LNetError: 323263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12019.047296] LNet: Removed LNI 192.168.201.105@tcp [12020.044171] Key type .llcrypt unregistered [12020.049277] Key type ._llcrypt unregistered [12038.187069] Key type ._llcrypt registered [12038.188664] Key type .llcrypt registered [12038.291358] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [12053.763610] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [12055.312453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12055.353268] alg: No test for adler32 (adler32-zlib) [12056.456270] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [12056.879288] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [12058.656913] Key type lgssc registered [12059.969362] Lustre: Echo OBD driver; http://www.lustre.org/ [12090.236600] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 12:56:05 (1786640165) [12097.994408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12098.035698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12099.327250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12099.350903] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12099.437538] Lustre: lustre-MDT0000: new disk, initializing [12099.553418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12099.568667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12103.879960] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12114.314616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12114.437698] Lustre: 326489:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12114.495645] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12114.498233] Lustre: Skipped 1 previous similar message [12114.566731] Lustre: lustre-MDT0001: new disk, initializing [12114.644264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12114.677405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12114.695946] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12119.087155] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12128.989566] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12134.262886] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12143.090329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12143.409413] Lustre: lustre-OST0000: new disk, initializing [12143.415567] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12143.427289] Lustre: 328458:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12143.544881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12150.360790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12150.375351] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12150.519673] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12150.526265] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12159.072409] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12176.360368] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12176.364514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12176.367513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12176.376051] Lustre: Skipped 1 previous similar message [12176.385552] Lustre: Skipped 1 previous similar message [12177.993840] Lustre: server umount lustre-OST0000 complete [12181.472765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12181.478661] LustreError: Skipped 1 previous similar message [12181.486804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12181.495593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12181.500118] Lustre: Skipped 1 previous similar message [12186.592496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12186.600476] Lustre: Skipped 2 previous similar messages [12187.830237] Lustre: server umount lustre-MDT0000 complete [12191.532289] LustreError: 328457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786640270 with bad export cookie 10209109452818486804 [12191.534744] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12191.550339] LustreError: 328457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12191.855390] Lustre: server umount lustre-MDT0001 complete [12202.171492] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12204.876314] Key type lgssc unregistered [12205.154163] LNet: 330367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12205.169532] LNetError: 330367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12205.197310] LNet: Removed LNI 192.168.201.105@tcp [12206.406518] Key type .llcrypt unregistered [12206.418895] Key type ._llcrypt unregistered [12229.965561] Key type ._llcrypt registered [12229.969212] Key type .llcrypt registered [12230.147309] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [12231.672988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12231.761071] alg: No test for adler32 (adler32-zlib) [12232.891218] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [12233.081436] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [12234.711682] Key type lgssc registered [12235.722798] Lustre: Echo OBD driver; http://www.lustre.org/ [12243.704819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12243.720443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12245.146269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12250.417311] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12254.690567] LustreError: 331641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12254.704110] LustreError: 331641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12259.789245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12259.814375] LustreError: 331642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12260.048197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12264.376675] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12272.191616] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12277.538316] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12283.931118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12284.261314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12290.702286] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12298.535368] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12311.957907] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12314.840867] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12321.105722] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.5@tcp (stopping) [12321.265524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12321.295604] Lustre: Skipped 2 previous similar messages [12326.184139] LustreError: 331638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12326.315573] Lustre: server umount lustre-MDT0000 complete [12330.478744] LustreError: 332378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786640409 with bad export cookie 4385225094040727070 [12330.489822] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12330.494089] LustreError: 332378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12331.311369] LustreError: 331636:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12331.343550] LustreError: 331636:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12331.366324] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.5@tcp (stopping) [12331.383578] Lustre: Skipped 3 previous similar messages [12331.490368] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12335.521269] LustreError: 331635:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12335.522296] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12335.547855] LustreError: 331635:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12335.569454] Lustre: Skipped 1 previous similar message [12337.147952] Lustre: server umount lustre-MDT0001 complete [12341.712397] Lustre: server umount lustre-OST0000 complete [12352.725355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12357.923703] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12360.544802] Lustre: 335825:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.105@tcp [12362.869796] Lustre: server umount MGS complete [12371.467460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12372.081742] LustreError: 336398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12372.107541] LustreError: 336398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12372.182292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12377.283500] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12386.938918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12387.431984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12392.322344] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12401.839246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12402.284433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12410.398051] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12419.353699] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12421.431216] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12427.916740] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12429.505370] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12435.887092] Lustre: server umount lustre-OST0000 complete [12442.599550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12442.604293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12442.614754] Lustre: Skipped 1 previous similar message [12442.623083] Lustre: Skipped 2 previous similar messages [12447.713037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12447.731227] Lustre: Skipped 1 previous similar message [12453.855201] Lustre: 330953:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786640516/real 1786640516] req@0000000021107bfa x1873428150810496/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786640532 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12453.858243] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12453.893913] Lustre: 330953:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [12455.391832] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12455.570719] Lustre: server umount lustre-MDT0000 complete [12458.975208] Lustre: 330952:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786640521/real 1786640521] req@00000000bf4bee5f x1873428150812288/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786640537 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12458.980725] LustreError: 336399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12459.060274] LustreError: 336399:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12459.601619] LustreError: 336379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786640538 with bad export cookie 4385225094040727910 [12459.604918] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12459.610878] LustreError: 336379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12460.021625] Lustre: server umount lustre-MDT0001 complete [12474.012305] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12478.386344] Key type lgssc unregistered [12478.642134] LNet: 339852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12478.665812] LNetError: 339852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12478.684916] LNet: Removed LNI 192.168.201.105@tcp [12479.542420] Key type .llcrypt unregistered [12479.546801] Key type ._llcrypt unregistered [12502.628329] Key type ._llcrypt registered [12502.629869] Key type .llcrypt registered [12502.777957] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [12521.045660] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [12522.473650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12522.562644] alg: No test for adler32 (adler32-zlib) [12523.617613] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [12523.871247] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [12525.584191] Key type lgssc registered [12526.948041] Lustre: Echo OBD driver; http://www.lustre.org/ [12566.306674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12566.331346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12567.735249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12567.787905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12567.927653] Lustre: lustre-MDT0000: new disk, initializing [12568.023346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12568.044188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12572.852817] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12585.361799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12585.493281] Lustre: 343017:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12585.538791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12585.542532] Lustre: Skipped 1 previous similar message [12585.625262] Lustre: lustre-MDT0001: new disk, initializing [12585.749810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12585.791690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12585.810893] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12591.402936] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12601.873284] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12607.232281] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12616.454267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12616.677395] Lustre: lustre-OST0000: new disk, initializing [12616.681832] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12616.690302] Lustre: 344985:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12616.792228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12622.401686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12622.413404] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12622.554762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12623.233648] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12633.758616] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12638.535349] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12638.737417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12643.271795] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12643.624525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12648.429362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12648.435393] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12648.439860] Lustre: Skipped 1 previous similar message [12648.440418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12653.550323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12653.560656] Lustre: Skipped 2 previous similar messages [12658.669489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12658.681495] Lustre: Skipped 1 previous similar message [12660.191148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12660.358637] Lustre: server umount lustre-OST0000 complete [12667.872633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12667.876827] LustreError: Skipped 1 previous similar message [12667.879259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12667.886862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12668.900491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12671.093929] Lustre: server umount lustre-MDT0000 complete [12675.714218] LustreError: 343008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786640754 with bad export cookie 6768558980422860128 [12675.731190] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12675.741774] LustreError: 343008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12676.088552] Lustre: server umount lustre-MDT0001 complete [12687.040553] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 13:06:02 (1786640762) [12704.702273] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [12716.721476] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [12758.837237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12759.107271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12759.122172] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12759.197545] Lustre: lustre-MDT0000: new disk, initializing [12759.259416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12759.283725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12764.006587] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12775.311054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12775.497484] Lustre: 349748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12775.626460] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12775.639262] Lustre: Skipped 1 previous similar message [12775.819602] Lustre: lustre-MDT0001: new disk, initializing [12776.025855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12776.096461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12776.115708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12782.537562] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12792.896079] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12799.995579] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12808.005295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12808.433646] Lustre: lustre-OST0000: new disk, initializing [12808.437203] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12808.447370] Lustre: 351715:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12808.610661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12814.431212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12814.440348] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12814.593396] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12815.805493] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12825.742557] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12860.900696] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12860.918169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12860.927210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12860.933072] Lustre: Skipped 1 previous similar message [12860.953631] Lustre: Skipped 3 previous similar messages [12871.141013] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12871.167553] Lustre: Skipped 3 previous similar messages [12874.207255] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12874.643980] Lustre: server umount lustre-OST0000 complete [12881.390903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12881.416972] Lustre: Skipped 1 previous similar message [12891.620046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12891.626526] Lustre: Skipped 5 previous similar messages [12893.151794] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12893.379379] Lustre: server umount lustre-MDT0000 complete [12896.736967] LustreError: 349760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12896.760701] LustreError: 349760:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12897.571156] LustreError: 349743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786640976 with bad export cookie 6768558980422861010 [12897.580076] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12897.586065] LustreError: 349743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12898.289732] Lustre: server umount lustre-MDT0001 complete [12910.374491] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12914.117539] Key type lgssc unregistered [12914.482080] LNet: 353898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12914.487444] LNetError: 353898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12914.508679] LNet: Removed LNI 192.168.201.105@tcp [12915.410387] Key type .llcrypt unregistered [12915.412810] Key type ._llcrypt unregistered [13020.414395] Key type ._llcrypt registered [13020.417529] Key type .llcrypt registered [13020.564179] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [13020.973332] Key type .llcrypt unregistered [13020.975568] Key type ._llcrypt unregistered [13103.495350] Key type ._llcrypt registered [13103.502405] Key type .llcrypt registered [13103.627568] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [13104.042419] Key type .llcrypt unregistered [13104.044631] Key type ._llcrypt unregistered [13136.623480] Key type ._llcrypt registered [13136.630165] Key type .llcrypt registered [13136.727472] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [13137.020656] Key type .llcrypt unregistered [13137.022664] Key type ._llcrypt unregistered [13165.073749] Key type ._llcrypt registered [13165.075718] Key type .llcrypt registered [13165.165697] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [13165.626566] Key type .llcrypt unregistered [13165.630702] Key type ._llcrypt unregistered [13251.215958] Key type ._llcrypt registered [13251.229993] Key type .llcrypt registered [13251.302794] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [13251.703463] Key type .llcrypt unregistered [13251.709530] Key type ._llcrypt unregistered [13279.996348] Key type ._llcrypt registered [13279.999863] Key type .llcrypt registered [13280.115229] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [13280.381359] Key type .llcrypt unregistered [13280.384091] Key type ._llcrypt unregistered [13313.688094] Key type ._llcrypt registered [13313.690417] Key type .llcrypt registered [13313.859646] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [13314.304280] Key type .llcrypt unregistered [13314.305940] Key type ._llcrypt unregistered [13350.015788] Key type ._llcrypt registered [13350.018508] Key type .llcrypt registered [13350.130202] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [13350.509442] Key type .llcrypt unregistered [13350.512305] Key type ._llcrypt unregistered [13382.936929] Key type ._llcrypt registered [13382.944959] Key type .llcrypt registered [13383.043668] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [13383.419714] Key type .llcrypt unregistered [13383.423530] Key type ._llcrypt unregistered [13422.490236] Key type ._llcrypt registered [13422.491921] Key type .llcrypt registered [13422.649704] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [13423.090279] Key type .llcrypt unregistered [13423.092562] Key type ._llcrypt unregistered [13491.472960] Key type ._llcrypt registered [13491.476906] Key type .llcrypt registered [13491.616785] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [13491.921722] Key type .llcrypt unregistered [13491.928294] Key type ._llcrypt unregistered [13536.553180] Key type ._llcrypt registered [13536.554451] Key type .llcrypt registered [13536.666404] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [13537.005414] Key type .llcrypt unregistered [13537.010891] Key type ._llcrypt unregistered [13575.596306] Key type ._llcrypt registered [13575.598476] Key type .llcrypt registered [13575.687694] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [13575.952283] Key type .llcrypt unregistered [13575.953992] Key type ._llcrypt unregistered [13631.491966] Key type ._llcrypt registered [13631.494897] Key type .llcrypt registered [13631.584754] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [13649.063564] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [13651.104666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13651.185714] alg: No test for adler32 (adler32-zlib) [13652.310508] Lustre: Lustre: Build Version: 2.17.57_2_g7a6d6aa [13652.600466] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [13654.311170] Key type lgssc registered [13655.851392] Lustre: Echo OBD driver; http://www.lustre.org/ [13692.748405] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13692.766763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13694.134647] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13694.172255] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13694.378753] Lustre: tst250fs-MDT0000: new disk, initializing [13694.493068] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13694.511814] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13699.933815] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13712.254234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13712.418879] Lustre: 374290:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13712.474737] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13712.478587] Lustre: Skipped 1 previous similar message [13712.535688] Lustre: tst250fs-MDT0001: new disk, initializing [13712.620746] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13712.680839] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13712.688700] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13717.711599] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13723.405412] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13730.745890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13731.141873] Lustre: tst250fs-OST0000: new disk, initializing [13731.147289] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13731.154657] Lustre: 375922:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13731.320524] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13734.073073] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13734.088326] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13734.281054] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13738.293520] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13749.228139] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13749.429840] Lustre: tst250fs-OST0001: new disk, initializing [13749.438104] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13749.448141] Lustre: 376792:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13749.561078] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13756.192748] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13759.109308] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13759.140759] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13759.242503] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13761.911140] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13764.888978] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13764.895649] Lustre: Skipped 2 previous similar messages [13784.544382] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13784.554242] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13784.572769] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13785.058147] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13785.083534] Lustre: Skipped 1 previous similar message [13785.091400] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13785.097692] Lustre: Skipped 3 previous similar messages [13789.971961] Lustre: server umount tst250fs-MDT0000 complete [13793.952262] LustreError: 374280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786641872 with bad export cookie 7914855330275796287 [13793.955897] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13793.961934] LustreError: 374280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13794.377663] Lustre: server umount tst250fs-MDT0001 complete [13808.876189] Lustre: server umount tst250fs-OST0000 complete [13810.785571] Lustre: 372517:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786641873/real 1786641873] req@ffff912d52923b80 x1873429639796224/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786641889 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13810.836672] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13810.850609] Lustre: Skipped 1 previous similar message [13813.232029] Lustre: server umount tst250fs-OST0001 complete [13834.108962] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [13845.312445] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [13884.374182] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 13:26:00 (1786641960) [13886.518674] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13905.020900] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [13916.893514] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [13952.474793] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13668 sec ========= 13:27:07 (1786642027) [13954.203613] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:27:10 (1786642030) === [13958.319498] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:27:13 (1786642033) === [14003.170973] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [14006.445969] Key type lgssc unregistered [14006.759416] LNet: 385947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14006.782070] LNetError: 385947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14007.848415] LNet: Removed LNI 192.168.201.105@tcp [14009.227152] Key type .llcrypt unregistered [14009.229357] Key type ._llcrypt unregistered