[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 473516086 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003268] x2apic enabled [ 0.004026] Switched APIC routing to physical x2apic. [ 0.005023] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009018] pid_max: default: 32768 minimum: 301 [ 0.010180] LSM: Security Framework initializing [ 0.011075] Yama: becoming mindful. [ 0.013049] SELinux: Initializing. [ 0.014077] *** VALIDATE selinux *** [ 0.023035] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028153] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029177] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030140] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031179] *** VALIDATE tmpfs *** [ 0.032586] *** VALIDATE proc *** [ 0.033330] *** VALIDATE cgroup *** [ 0.034018] *** VALIDATE cgroup2 *** [ 0.035328] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037095] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040041] Spectre V2 : User space: Vulnerable [ 0.041016] Speculative Store Bypass: Vulnerable [ 0.044560] debug: unmapping init [mem 0xffffffffaa859000-0xffffffffaa860fff] [ 0.047193] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048863] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049036] ... version: 2 [ 0.050023] ... bit width: 48 [ 0.051024] ... generic registers: 4 [ 0.052022] ... value mask: 0000ffffffffffff [ 0.053023] ... max period: 00007fffffffffff [ 0.054024] ... fixed-purpose events: 3 [ 0.055031] ... event mask: 000000070000000f [ 0.056414] rcu: Hierarchical SRCU implementation. [ 0.058870] smp: Bringing up secondary CPUs ... [ 0.059676] x86: Booting SMP configuration: [ 0.060042] .... node #0, CPUs: #1 #2 #3 [ 0.064019] smp: Brought up 1 node, 4 CPUs [ 0.066030] smpboot: Max logical packages: 1 [ 0.067021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.232711] node 0 deferred pages initialised in 163ms [ 0.237304] devtmpfs: initialized [ 0.238217] x86/mm: Memory block size: 128MB [ 0.240363] gcov: version magic: 0x41383552 [ 0.242283] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.244119] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.246280] pinctrl core: initialized pinctrl subsystem [ 0.247254] [ 0.247796] ************************************************************* [ 0.248031] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.249021] ** ** [ 0.250023] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.251019] ** ** [ 0.252022] ** This means that this kernel is built to expose internal ** [ 0.253020] ** IOMMU data structures, which may compromise security on ** [ 0.254020] ** your system. ** [ 0.255020] ** ** [ 0.256035] ** If you see this message and you are not debugging the ** [ 0.257022] ** kernel, report this immediately to your vendor! ** [ 0.258020] ** ** [ 0.259021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.260021] ************************************************************* [ 0.261761] NET: Registered protocol family 16 [ 0.262540] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.263088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.264088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.266062] cpuidle: using governor menu [ 0.268985] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.271556] PCI: Using configuration type 1 for base access [ 0.274136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.284167] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.285027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.287149] cryptd: max_cpu_qlen set to 1000 [ 0.290015] ACPI: Added _OSI(Module Device) [ 0.291024] ACPI: Added _OSI(Processor Device) [ 0.292022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.293023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.297141] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.300512] ACPI: Interpreter enabled [ 0.301082] ACPI: PM: (supports S0 S3 S4 S5) [ 0.302018] ACPI: Using IOAPIC for interrupt routing [ 0.303146] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.304508] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.313775] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.314072] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.315034] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.316156] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.318621] acpiphp: Slot [2] registered [ 0.319217] acpiphp: Slot [5] registered [ 0.320231] acpiphp: Slot [6] registered [ 0.321252] acpiphp: Slot [7] registered [ 0.322211] acpiphp: Slot [8] registered [ 0.323200] acpiphp: Slot [9] registered [ 0.324237] acpiphp: Slot [10] registered [ 0.325189] acpiphp: Slot [3] registered [ 0.326248] acpiphp: Slot [4] registered [ 0.327222] acpiphp: Slot [11] registered [ 0.328158] acpiphp: Slot [12] registered [ 0.329161] acpiphp: Slot [13] registered [ 0.330136] acpiphp: Slot [14] registered [ 0.331152] acpiphp: Slot [15] registered [ 0.332152] acpiphp: Slot [16] registered [ 0.333158] acpiphp: Slot [17] registered [ 0.334177] acpiphp: Slot [18] registered [ 0.335144] acpiphp: Slot [19] registered [ 0.336136] acpiphp: Slot [20] registered [ 0.337118] acpiphp: Slot [21] registered [ 0.338139] acpiphp: Slot [22] registered [ 0.339162] acpiphp: Slot [23] registered [ 0.340178] acpiphp: Slot [24] registered [ 0.341149] acpiphp: Slot [25] registered [ 0.342139] acpiphp: Slot [26] registered [ 0.343144] acpiphp: Slot [27] registered [ 0.344150] acpiphp: Slot [28] registered [ 0.345239] acpiphp: Slot [29] registered [ 0.346152] acpiphp: Slot [30] registered [ 0.347198] acpiphp: Slot [31] registered [ 0.348091] PCI host bridge to bus 0000:00 [ 0.349033] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.350045] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.351029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.352036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.353040] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.354042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.355238] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.357032] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.359503] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.365022] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.369021] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.370031] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.371031] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.372029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.373855] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.374854] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.375072] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.376902] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.379024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.389927] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.391025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.395796] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.398035] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.401029] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.408027] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.415246] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.418032] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.421033] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.429024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.434865] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.444020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.451021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.463043] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.474043] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.482022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.489021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.508021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.518231] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.523022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.529030] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.542021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.550596] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.572019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.580032] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.604023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.616443] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.618373] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.620390] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.623389] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.625271] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.629145] iommu: Default domain type: Passthrough [ 0.631284] SCSI subsystem initialized [ 0.632150] ACPI: bus type USB registered [ 0.634117] usbcore: registered new interface driver usbfs [ 0.635099] usbcore: registered new interface driver hub [ 0.637083] usbcore: registered new device driver usb [ 0.638182] pps_core: LinuxPPS API ver. 1 registered [ 0.640013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.642077] PTP clock support registered [ 0.646386] EDAC MC: Ver: 3.0.0 [ 0.647514] PCI: Using ACPI for IRQ routing [ 0.649041] NetLabel: Initializing [ 0.650031] NetLabel: domain hash size = 128 [ 0.651012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.652168] NetLabel: unlabeled traffic allowed by default [ 0.654198] vgaarb: loaded [ 0.655331] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.657013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.664178] clocksource: Switched to clocksource kvm-clock [ 0.773677] VFS: Disk quotas dquot_6.6.0 [ 0.775312] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.777887] *** VALIDATE ramfs *** [ 0.779035] *** VALIDATE hugetlbfs *** [ 0.780529] pnp: PnP ACPI init [ 0.783581] pnp: PnP ACPI: found 6 devices [ 0.826665] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.830818] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.833441] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.835915] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.838435] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.841317] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.845719] NET: Registered protocol family 2 [ 0.848673] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.854176] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.858156] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.863825] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.867480] TCP: Hash tables configured (established 65536 bind 65536) [ 0.870315] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.873687] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.876866] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.880373] NET: Registered protocol family 1 [ 0.883608] RPC: Registered named UNIX socket transport module. [ 0.886087] RPC: Registered udp transport module. [ 0.887980] RPC: Registered tcp transport module. [ 0.889812] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.892198] NET: Registered protocol family 44 [ 0.893846] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.895888] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.897777] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.900061] PCI: CLS 0 bytes, default 64 [ 0.901689] Unpacking initramfs... [ 2.268735] debug: unmapping init [mem 0xffff8a357cc54000-0xffff8a357ffbffff] [ 2.272790] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.275093] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.277727] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.803108] Initialise system trusted keyrings [ 2.805030] Key type blacklist registered [ 2.807053] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.817395] zbud: loaded [ 2.820801] *** VALIDATE nfs *** [ 2.821736] *** VALIDATE nfs4 *** [ 2.823332] pstore: using deflate compression [ 2.826374] Platform Keyring initialized [ 2.938882] NET: Registered protocol family 38 [ 2.940302] Key type asymmetric registered [ 2.941665] Asymmetric key parser 'x509' registered [ 2.943271] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.946324] io scheduler mq-deadline registered [ 2.947479] io scheduler kyber registered [ 2.948623] io scheduler bfq registered [ 2.949912] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.951958] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.954645] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.957412] ACPI: Power Button [PWRF] [ 2.961878] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.968684] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.980717] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.988333] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.998933] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.025178] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.051834] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.056276] Non-volatile memory driver v1.3 [ 3.057592] Linux agpgart interface v0.103 [ 3.089933] virtio_blk virtio1: [vda] 149984 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.093267] vda: detected capacity change from 0 to 76791808 [ 3.116975] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.120424] vdb: detected capacity change from 0 to 1073741824 [ 3.136934] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.139219] vdc: detected capacity change from 0 to 2621440000 [ 3.155478] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.159154] vdd: detected capacity change from 0 to 2621440000 [ 3.175699] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.178052] vde: detected capacity change from 0 to 4294967296 [ 3.192593] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.194543] vdf: detected capacity change from 0 to 4294967296 [ 3.199786] libphy: Fixed MDIO Bus: probed [ 3.214586] usbcore: registered new interface driver usbserial_generic [ 3.217632] usbserial: USB Serial support registered for generic [ 3.220255] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.224257] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.226064] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.228729] mousedev: PS/2 mouse device common for all mice [ 3.231687] rtc_cmos 00:05: RTC can wake from S4 [ 3.235125] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.235198] rtc_cmos 00:05: registered as rtc0 [ 3.240651] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.242548] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.245963] intel_pstate: CPU model not supported [ 3.248279] hid: raw HID events driver (C) Jiri Kosina [ 3.256375] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.261128] usbcore: registered new interface driver usbhid [ 3.261136] usbhid: USB HID core driver [ 3.261286] drop_monitor: Initializing network drop monitor service [ 3.261420] Initializing XFRM netlink socket [ 3.261811] NET: Registered protocol family 10 [ 3.263029] Segment Routing with IPv6 [ 3.273592] NET: Registered protocol family 17 [ 3.275328] mpls_gso: MPLS GSO support [ 3.281086] RAS: Correctable Errors collector initialized. [ 3.282406] AVX version of gcm_enc/dec engaged. [ 3.283444] AES CTR mode by8 optimization enabled [ 3.338439] sched_clock: Marking stable (3338418975, 0)->(4517027675, -1178608700) [ 3.340686] registered taskstats version 1 [ 3.342183] Loading compiled-in X.509 certificates [ 3.343892] zswap: loaded using pool lzo/zbud [ 3.364623] Key type big_key registered [ 3.377258] Key type encrypted registered [ 3.378939] ima: No TPM chip found, activating TPM-bypass! [ 3.381015] ima: Allocated hash algorithm: sha1 [ 3.382660] ima: No architecture policies found [ 3.384420] evm: Initialising EVM extended attributes: [ 3.386360] evm: security.selinux [ 3.387559] evm: security.ima [ 3.388612] evm: security.capability [ 3.389869] evm: HMAC attrs: 0x1 [ 3.392152] rtc_cmos 00:05: setting system clock to 2026-09-16 17:09:17 UTC (1789578557) [ 3.397841] debug: unmapping init [mem 0xffffffffab803000-0xffffffffab9fffff] [ 3.400584] debug: unmapping init [mem 0xffffffffaa582000-0xffffffffaa858fff] [ 3.413083] Write protecting the kernel read-only data: 28672k [ 3.416533] debug: unmapping init [mem 0xffffffffa8c03000-0xffffffffa8dfffff] [ 3.419419] debug: unmapping init [mem 0xffffffffa9514000-0xffffffffa95fffff] [ 3.454586] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.463675] systemd[1]: Detected virtualization kvm. [ 3.465658] systemd[1]: Detected architecture x86-64. [ 3.467612] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.494856] systemd[1]: No hostname configured. [ 3.496714] systemd[1]: Set hostname to . [ 3.498871] random: systemd: uninitialized urandom read (16 bytes read) [ 3.501225] systemd[1]: Initializing machine ID from random generator. [ 3.625563] random: systemd: uninitialized urandom read (16 bytes read) [ 3.628096] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.632804] random: systemd: uninitialized urandom read (16 bytes read) [ 3.635234] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.639610] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.229484] device-mapper: uevent: version 1.0.3 [ 4.231985] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.924776] random: fast init done [ 4.946593] virtio_net virtio0 ens2: renamed from eth0 [ 4.984824] scsi host0: ata_piix [ 5.019372] scsi host1: ata_piix [ 5.021207] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.023944] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.462906] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.899912] random: crng init done [ 9.900868] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.307498] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.461287] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.701096] SELinux: Disabled at runtime. [ 11.759098] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.767752] systemd[1]: Detected virtualization kvm. [ 11.769647] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.235484] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.238839] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.243721] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.247371] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.250576] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.259356] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.263392] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ 12.304560] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 12.693387] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.985489] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.013909] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.144530] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.162465] EDAC sbridge: Ver: 1.1.2 [ 15.036792] Key type dns_resolver registered [ 15.340280] NFS: Registering the id_resolver key type [ 15.342171] Key type id_resolver registered [ 15.343890] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg148-server login: [ 63.042360] hrtimer: interrupt took 4518281 ns [ 104.541035] libcfs: loading out-of-tree module taints kernel. [ 104.590927] Key type ._llcrypt registered [ 104.593956] Key type .llcrypt registered [ 104.738804] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 124.765422] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 127.066513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 127.095778] alg: No test for adler32 (adler32-zlib) [ 128.739347] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 129.712273] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 131.593640] Key type lgssc registered [ 133.585134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 150.375605] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 191.924599] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 206.052414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 206.096103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 207.469533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 207.506525] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 207.600239] Lustre: lustre-MDT0000: new disk, initializing [ 207.668976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 207.683433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 211.082850] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.375768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 227.589463] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 227.718838] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 227.726421] Lustre: Skipped 1 previous similar message [ 227.892453] Lustre: lustre-MDT0001: new disk, initializing [ 228.026432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 228.095491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 228.112961] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 232.850720] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.634639] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 249.254788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.611960] Lustre: lustre-OST0000: new disk, initializing [ 249.615716] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 249.655658] Lustre: 8452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 249.763375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 255.030355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 255.044509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 255.168553] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 256.931317] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 271.848824] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 272.035656] Lustre: lustre-OST0001: new disk, initializing [ 272.038966] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 272.044479] Lustre: 9525:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 272.111778] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 278.053412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 278.064983] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 278.113711] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 278.812345] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 290.697683] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 299.007959] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 305.802560] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing check_logdir /tmp/testlogs/ [ 310.928328] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing yml_node [ 314.658245] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 317.547468] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 320.571063] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 322.667428] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 16 13:14:33 EDT 2026 [ 347.862715] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 351.064594] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 370.144772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 370.160754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 370.162666] Lustre: Skipped 1 previous similar message [ 370.184029] Lustre: Skipped 2 previous similar messages [ 373.198381] Lustre: server umount lustre-MDT0000 complete [ 377.089713] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789578931 with bad export cookie 3045418725470845840 [ 377.091666] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 377.101045] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 380.384456] LustreError: 10372:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 380.388377] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 380.423238] LustreError: 10372:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 380.453076] Lustre: Skipped 2 previous similar messages [ 380.458449] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 380.475209] Lustre: Skipped 1 previous similar message [ 382.435564] LustreError: 10372:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 382.436475] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 382.471349] Lustre: Skipped 1 previous similar message [ 383.759213] Lustre: server umount lustre-MDT0001 complete [ 387.711507] Lustre: server umount lustre-OST0000 complete [ 391.553676] Lustre: server umount lustre-OST0001 complete [ 399.602801] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 408.485989] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 447.755572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 448.059109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 448.108746] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 448.239702] Lustre: lustre-MDT0000: new disk, initializing [ 448.313505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 448.348182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 453.581134] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 465.453771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.680608] Lustre: 15081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 465.841272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 465.853376] Lustre: Skipped 1 previous similar message [ 466.009339] Lustre: lustre-MDT0001: new disk, initializing [ 466.297908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 466.379687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 466.416382] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 470.949354] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 479.817388] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 485.716847] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 492.642644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 493.041647] Lustre: lustre-OST0000: new disk, initializing [ 493.047868] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 493.060538] Lustre: 17052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 493.143187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 494.303759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 494.328784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 494.443666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 499.399101] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 509.854770] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 515.598689] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 515.845701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 520.589938] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 520.850244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 525.304444] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 525.305352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 525.312199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 525.320618] LustreError: Skipped 1 previous similar message [ 525.351186] Lustre: Skipped 2 previous similar messages [ 529.274487] Lustre: server umount lustre-OST0000 complete [ 532.960569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 532.977167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 532.998563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 533.002597] Lustre: Skipped 1 previous similar message [ 545.762693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 545.775072] Lustre: Skipped 5 previous similar messages [ 547.296988] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 547.484816] Lustre: server umount lustre-MDT0000 complete [ 550.883887] LustreError: 15093:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.903563] LustreError: 15093:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 551.480392] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789579105 with bad export cookie 3045418725470847373 [ 551.486343] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 551.492777] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 551.865798] Lustre: server umount lustre-MDT0001 complete [ 558.743866] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 13:18:30 (1789579110) [ 581.658958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 582.493182] LustreError: 19927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 582.753455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 587.762109] LustreError: 19928:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 588.610365] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 596.962065] LustreError: 19927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 596.976996] LustreError: 19927:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 601.309382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 607.190913] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 616.416255] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 622.692735] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 625.329064] Lustre: Failing over lustre-MDT0000 [ 625.744215] Lustre: server umount lustre-MDT0000 complete [ 626.403900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 626.417696] Lustre: Skipped 1 previous similar message [ 626.428627] LustreError: 19924:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 626.449625] LustreError: 19924:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 642.015526] Lustre: 3646:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789579180/real 1789579180] req@ffff8a35c595df80 x1876509257272576/t0(0) o400->MGC192.168.201.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789579196 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 642.045080] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 646.215194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 653.280208] LustreError: 3644:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a34c3347b80 x1876509257275520/t0(0) o250->MGC192.168.201.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 653.588252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 653.598952] Lustre: Skipped 1 previous similar message [ 653.666575] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 658.913326] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 658.926373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 658.985119] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 659.070973] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 671.431593] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 676.089827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 676.101370] Lustre: Skipped 1 previous similar message [ 676.103859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 682.706563] Lustre: server umount lustre-MDT0000 complete [ 686.566490] LustreError: 19927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 686.593093] LustreError: 19927:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 686.844346] LustreError: 19909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789579240 with bad export cookie 3045418725470848787 [ 686.856100] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 686.870288] LustreError: 19909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 687.272855] Lustre: server umount lustre-MDT0001 complete [ 697.228943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 702.791152] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 711.575607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 716.825352] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 725.733719] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 732.594107] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 735.160152] Lustre: Failing over lustre-MDT0000 [ 735.497327] Lustre: server umount lustre-MDT0000 complete [ 736.225505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 736.232575] Lustre: Skipped 1 previous similar message [ 751.584575] LustreError: 23265:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 751.624289] LustreError: 23265:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 752.607290] Lustre: 3648:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789579290/real 1789579290] req@ffff8a34c3347b80 x1876509257303424/t0(0) o400->MGC192.168.201.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789579306 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 752.628186] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 755.953473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 764.263090] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 769.507583] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 769.511837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 769.534853] Lustre: Skipped 1 previous similar message [ 769.576675] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 769.704790] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 781.876337] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 786.921352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 786.927568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 786.940556] Lustre: Skipped 1 previous similar message [ 786.956396] Lustre: Skipped 4 previous similar messages [ 792.863658] Lustre: server umount lustre-MDT0000 complete [ 796.745480] LustreError: 25934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789579350 with bad export cookie 3045418725470849725 [ 796.751551] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 796.755767] LustreError: 25934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 797.235996] Lustre: server umount lustre-MDT0001 complete [ 807.567676] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 13:22:39 (1789579359) [ 825.603804] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 837.090147] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 887.585855] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 901.969917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 902.223830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 902.254953] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 902.317420] Lustre: lustre-MDT0000: new disk, initializing [ 902.375919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 902.381289] Lustre: Skipped 3 previous similar messages [ 902.392680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 907.033883] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 919.474043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 919.682365] Lustre: 29651:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 919.943269] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 925.426079] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 937.535230] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 945.441403] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 954.582207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 954.992504] Lustre: 31619:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 956.955290] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 957.002168] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 957.094765] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 957.137476] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 961.779450] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 972.440195] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 979.028392] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1002.977737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1002.986440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1002.999605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1003.013821] Lustre: Skipped 2 previous similar messages [ 1008.622295] LustreError: 31603:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1008.646109] LustreError: 31603:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 1008.831129] Lustre: server umount lustre-OST0000 complete [ 1020.207396] Lustre: server umount lustre-MDT0000 complete [ 1026.129763] LustreError: 30444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789579580 with bad export cookie 3045418725470850131 [ 1026.136169] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1026.145805] LustreError: 30444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1026.640437] Lustre: server umount lustre-MDT0001 complete [ 1040.792906] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 1044.443641] Key type lgssc unregistered [ 1044.910362] LNet: 33596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.917138] LNetError: 33596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1044.939936] LNet: Removed LNI 192.168.201.148@tcp [ 1046.101088] Key type .llcrypt unregistered [ 1046.103771] Key type ._llcrypt unregistered [ 1076.250197] Key type ._llcrypt registered [ 1076.255203] Key type .llcrypt registered [ 1076.437871] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 1077.507390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1077.574063] alg: No test for adler32 (adler32-zlib) [ 1078.787766] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 1079.000579] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 1080.703604] Key type lgssc registered [ 1082.163894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1091.525991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1091.548841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1093.034472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1098.102751] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1103.335298] LustreError: 35011:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1103.366939] LustreError: 35011:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1107.550954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1108.091507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1113.476525] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1124.092165] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1132.460485] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1140.905540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1141.284055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1148.342665] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1153.521534] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1156.336354] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1163.845647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1184.577573] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 13:28:55 (1789579735) [ 1186.752157] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1188.839884] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1196.257128] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 13:29:07 (1789579747) [ 1205.808411] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 13:29:17 (1789579757) [ 1207.198886] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1217.084516] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1217.097639] Lustre: Skipped 2 previous similar messages [ 1219.016906] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 13:29:30 (1789579770) [ 1221.605524] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1221.610228] Lustre: Skipped 2 previous similar messages [ 1230.284705] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1230.289324] Lustre: Skipped 4 previous similar messages [ 1248.570357] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 13:29:58 (1789579798) [ 1252.456034] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1257.244789] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1268.175439] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 13:30:19 (1789579819) [ 1270.496683] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1270.502612] Lustre: Skipped 6 previous similar messages [ 1288.451826] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 13:30:38 (1789579838) [ 1305.060728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1305.065617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1305.076777] Lustre: Skipped 1 previous similar message [ 1305.083104] Lustre: Skipped 1 previous similar message [ 1310.178377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1310.183044] Lustre: Skipped 1 previous similar message [ 1315.299058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1315.306113] Lustre: Skipped 1 previous similar message [ 1319.391117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1319.554462] Lustre: server umount lustre-OST0000 complete [ 1325.545703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.585443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1325.603456] Lustre: Skipped 1 previous similar message [ 1329.820534] Lustre: server umount lustre-MDT0000 complete [ 1330.666035] LustreError: 35008:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1330.705082] LustreError: 35008:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1333.961889] LustreError: 34993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789579888 with bad export cookie 5479884102341465603 [ 1333.975719] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1333.980502] LustreError: 34993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1334.334615] Lustre: server umount lustre-MDT0001 complete [ 1348.294589] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 1352.231729] Key type lgssc unregistered [ 1352.593403] LNet: 42891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1352.601229] LNetError: 42891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1353.639718] LNet: Removed LNI 192.168.201.148@tcp [ 1354.561954] Key type .llcrypt unregistered [ 1354.567882] Key type ._llcrypt unregistered [ 1374.013656] Key type ._llcrypt registered [ 1374.016899] Key type .llcrypt registered [ 1374.153418] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 1394.551317] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 1395.805827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1395.991386] alg: No test for adler32 (adler32-zlib) [ 1397.388319] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 1397.709223] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 1399.418716] Key type lgssc registered [ 1400.896465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1446.304713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1446.322659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1447.623841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1447.688118] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1447.817250] Lustre: lustre-MDT0000: new disk, initializing [ 1447.935120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1447.964785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1452.427852] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1463.156924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1463.303450] Lustre: 46048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1463.417576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1463.425411] Lustre: Skipped 1 previous similar message [ 1463.532719] Lustre: lustre-MDT0001: new disk, initializing [ 1463.627783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1463.660547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1463.672804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1468.092107] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1478.287797] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1486.417162] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1494.488714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1494.809496] Lustre: lustre-OST0000: new disk, initializing [ 1494.812162] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1494.815240] Lustre: 48014:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1494.917360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1500.556438] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1502.841604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1502.847792] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1502.951061] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1509.837746] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1515.071717] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1515.291742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1521.529907] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1521.780535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1528.810116] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1528.821987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1528.825419] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1530.913799] Lustre: server umount lustre-OST0000 complete [ 1535.457880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1535.477742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1535.518229] Lustre: Skipped 1 previous similar message [ 1535.527055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1535.535845] Lustre: Skipped 1 previous similar message [ 1539.050497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1539.052630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1544.163820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1544.185700] Lustre: Skipped 2 previous similar messages [ 1549.279434] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1549.280638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1549.292215] Lustre: Skipped 1 previous similar message [ 1549.483604] Lustre: server umount lustre-MDT0000 complete [ 1553.638034] LustreError: 46040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789580107 with bad export cookie 5887544814588925198 [ 1553.645567] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1554.246140] Lustre: server umount lustre-MDT0001 complete [ 1566.088729] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 13:35:17 (1789580117) [ 1573.829537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1574.749986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1580.001617] LustreError: 50475:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1580.027350] LustreError: 50475:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1580.595426] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1584.106535] LustreError: 50476:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1589.218023] LustreError: 50475:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1592.529877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1593.284645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1600.604753] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1610.524280] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1617.584231] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1627.662486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1628.151617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1634.582195] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1642.266693] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1651.995297] LustreError: 53005:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1653.121768] Lustre: Modifying parameter general.timeout=1 in log params [ 1655.272385] Lustre: Modifying parameter general.timeout=3 in log params [ 1655.278460] Lustre: Skipped 1 previous similar message [ 1657.968125] Lustre: Modifying parameter general.timeout=6 in log params [ 1657.974817] Lustre: Skipped 2 previous similar messages [ 1662.142793] Lustre: Modifying parameter general.timeout=10 in log params [ 1662.148745] Lustre: Skipped 3 previous similar messages [ 1670.451426] Lustre: Modifying parameter general.timeout=19 in log params [ 1670.458261] Lustre: Skipped 8 previous similar messages [ 1686.503366] Lustre: Modifying parameter general.timeout=34 in log params [ 1686.505667] Lustre: Skipped 14 previous similar messages [ 1718.745440] Lustre: Modifying parameter general.timeout=63 in log params [ 1718.763569] Lustre: Skipped 28 previous similar messages [ 1792.784414] Lustre: Disabling parameter general.timeout= in log params [ 1792.788875] Lustre: Skipped 66 previous similar messages [ 1797.226423] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 13:39:08 (1789580348) [ 1801.227093] Lustre: 59962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1803.552563] Lustre: 60061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1806.886772] Lustre: 60277:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1806.893519] Lustre: 60277:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1840.293931] Lustre: 61844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1840.315910] Lustre: 61844:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1840.347177] LustreError: 61844:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1844.817290] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 13:39:56 (1789580396) [ 1858.017037] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1858.020568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1858.030192] Lustre: Skipped 1 previous similar message [ 1862.348429] Lustre: server umount lustre-MDT0000 complete [ 1863.146552] LustreError: 50475:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1865.874055] LustreError: 50456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789580419 with bad export cookie 5887544814588926080 [ 1865.884098] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1865.886756] LustreError: 50456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1866.224729] Lustre: server umount lustre-MDT0001 complete [ 1880.919744] Lustre: server umount lustre-OST0000 complete [ 1906.960402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1907.460933] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1907.860226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1912.618835] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1921.765354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1921.946632] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1921.980647] Lustre: 64375:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1926.646518] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1934.669585] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1942.142189] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1950.162602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1950.513701] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1950.978374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1950.982939] Lustre: Skipped 1 previous similar message [ 1959.270648] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1965.793621] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1967.935120] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1967.946630] Lustre: Skipped 23 previous similar messages [ 1968.033496] Lustre: 66939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1976.812875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1976.826792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1976.869523] Lustre: Skipped 1 previous similar message [ 1976.893119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1976.919548] Lustre: Skipped 2 previous similar messages [ 1981.058727] Lustre: server umount lustre-OST0000 complete [ 1987.554953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1987.576777] Lustre: Skipped 2 previous similar messages [ 2000.352871] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2000.560262] Lustre: server umount lustre-MDT0000 complete [ 2002.913674] LustreError: 64380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2002.940095] LustreError: 64380:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2004.441447] LustreError: 64366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789580558 with bad export cookie 5887544814588929258 [ 2004.443895] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2004.447635] LustreError: 64366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2004.737898] Lustre: server umount lustre-MDT0001 complete [ 2017.547450] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 2022.140460] Key type lgssc unregistered [ 2022.553772] LNet: 68148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2022.565698] LNetError: 68148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2023.592413] LNet: Removed LNI 192.168.201.148@tcp [ 2024.972907] Key type .llcrypt unregistered [ 2024.974677] Key type ._llcrypt unregistered [ 2053.926356] Key type ._llcrypt registered [ 2053.929319] Key type .llcrypt registered [ 2054.096673] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2055.188384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2055.224823] alg: No test for adler32 (adler32-zlib) [ 2056.408648] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 2056.650601] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 2058.344201] Key type lgssc registered [ 2059.683738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2067.474569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2067.516782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2069.174265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2073.752483] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2079.713517] LustreError: 69558:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2079.764916] LustreError: 69558:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2083.045861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2083.752453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2089.268433] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2097.415477] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2103.881870] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2111.986939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2112.752776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2119.353465] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2127.095389] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2135.979302] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2148.848747] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2148.850946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2148.861989] Lustre: Skipped 1 previous similar message [ 2151.675914] Lustre: server umount lustre-MDT0000 complete [ 2153.956744] LustreError: 69554:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2153.988108] LustreError: 69554:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2154.709767] LustreError: 69539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789580708 with bad export cookie 1281585946490383175 [ 2154.714240] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2154.715615] LustreError: 69539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2154.940827] Lustre: server umount lustre-MDT0001 complete [ 2168.482838] Lustre: server umount lustre-OST0000 complete [ 2203.249479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2203.782913] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2203.851615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2204.364293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2209.850786] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2220.092242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2220.245824] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2220.274370] Lustre: 74432:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2220.901290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2227.077734] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2234.972936] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2241.762742] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2252.457774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2252.810557] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2253.224116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2262.646321] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2272.110811] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2274.107865] Lustre: 77008:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2276.014343] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2282.977801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2282.984188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2282.992652] Lustre: Skipped 1 previous similar message [ 2282.996367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2283.000197] Lustre: Skipped 2 previous similar messages [ 2283.498281] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2288.612775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2288.625511] Lustre: Skipped 2 previous similar messages [ 2288.825373] Lustre: server umount lustre-OST0000 complete [ 2293.730336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2293.734851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2293.744063] Lustre: Skipped 2 previous similar messages [ 2293.758019] Lustre: Skipped 2 previous similar messages [ 2298.851403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2298.854540] Lustre: Skipped 1 previous similar message [ 2299.827539] Lustre: server umount lustre-MDT0000 complete [ 2303.976694] LustreError: 74444:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2304.005330] LustreError: 74444:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2304.027840] LustreError: 74423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789580858 with bad export cookie 1281585946490384022 [ 2304.033878] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2304.035765] LustreError: 74423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2305.086627] Lustre: server umount lustre-MDT0001 complete [ 2318.173126] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 2320.296739] Key type lgssc unregistered [ 2320.524703] LNet: 78220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2320.529990] LNetError: 78220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2320.542425] LNet: Removed LNI 192.168.201.148@tcp [ 2321.327674] Key type .llcrypt unregistered [ 2321.328764] Key type ._llcrypt unregistered [ 2358.272770] Key type ._llcrypt registered [ 2358.276318] Key type .llcrypt registered [ 2358.395858] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2359.805328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2359.933429] alg: No test for adler32 (adler32-zlib) [ 2361.029503] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 2361.242143] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 2362.903202] Key type lgssc registered [ 2364.570343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2373.789181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2373.847743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2375.654848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2381.140384] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2385.894719] LustreError: 79631:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2385.921500] LustreError: 79631:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2390.881262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2391.017154] LustreError: 79632:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2391.536134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2396.544987] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2404.133875] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2412.073301] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2420.120227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2420.671744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2427.074716] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2436.097342] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2443.666439] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2444.578511] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2445.931825] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2448.152097] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2448.158614] Lustre: Skipped 1 previous similar message [ 2452.790816] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2452.796907] Lustre: Skipped 4 previous similar messages [ 2461.008490] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2461.010976] Lustre: Skipped 8 previous similar messages [ 2477.615059] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2477.634990] Lustre: Skipped 16 previous similar messages [ 2509.755079] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2509.759362] Lustre: Skipped 30 previous similar messages [ 2554.141367] Lustre: 87745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2557.093451] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 13:51:48 (1789581108) [ 2559.012632] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2560.985812] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 13:51:52 (1789581112) [ 2569.184852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2569.187722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2569.195190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2569.212452] Lustre: Skipped 1 previous similar message [ 2572.011297] Lustre: server umount lustre-OST0000 complete [ 2579.427953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2579.435628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2579.470435] Lustre: Skipped 1 previous similar message [ 2579.477272] Lustre: Skipped 1 previous similar message [ 2583.206687] Lustre: server umount lustre-MDT0000 complete [ 2584.545062] LustreError: 79631:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.562311] LustreError: 79631:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2587.653858] LustreError: 79611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789581141 with bad export cookie 10940405983736620782 [ 2587.662657] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2587.689372] LustreError: 79611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2588.214873] Lustre: server umount lustre-MDT0001 complete [ 2599.665362] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 2601.895920] Key type lgssc unregistered [ 2602.069414] LNet: 88993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2602.071873] LNetError: 88993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2602.087595] LNet: Removed LNI 192.168.201.148@tcp [ 2602.636121] Key type .llcrypt unregistered [ 2602.643060] Key type ._llcrypt unregistered [ 2610.366745] Key type ._llcrypt registered [ 2610.374261] Key type .llcrypt registered [ 2610.584168] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2618.192896] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules [ 2619.848512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2619.863630] alg: No test for adler32 (adler32-zlib) [ 2621.294881] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 2621.839879] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 2623.715361] Key type lgssc registered [ 2625.608479] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2627.112055] LustreError: 90239:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2647.643409] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2649.872883] LustreError: 90239:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2657.549659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2657.566074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2659.247277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2664.952273] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2669.551186] LustreError: 91244:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2669.573619] LustreError: 91244:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2674.668265] LustreError: 91245:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.938927] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 13:53:48 (1789581228) [ 2679.793362] LustreError: 91244:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2682.812421] LustreError: 91275:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2682.823739] LustreError: 91275:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2682.827711] LustreError: 91275:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 2683.550620] Lustre: server umount lustre-MDT0000 complete [ 2700.134946] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 2703.324764] Key type lgssc unregistered [ 2703.673834] LNet: 92767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2703.684207] LNetError: 92767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2704.743566] LNet: Removed LNI 192.168.201.148@tcp [ 2705.621732] Key type .llcrypt unregistered [ 2705.626858] Key type ._llcrypt unregistered [ 2729.479529] Key type ._llcrypt registered [ 2729.483094] Key type .llcrypt registered [ 2729.660373] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2731.464795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2731.550259] alg: No test for adler32 (adler32-zlib) [ 2732.724893] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 2733.021103] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 2734.743366] Key type lgssc registered [ 2735.842031] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2744.995142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2745.021449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2746.768961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2750.983209] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2756.072269] LustreError: 93985:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2756.108201] LustreError: 93985:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2761.212115] LustreError: 93986:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2761.682262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2762.866479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2769.485280] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2781.787256] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2789.626426] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2797.882584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2798.598426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2805.669703] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2814.951850] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2877.600146] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 13:57:08 (1789581428) [ 2880.051818] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2881.893923] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 13:57:13 (1789581433) [ 2887.651207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2887.658381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2887.687781] Lustre: Skipped 2 previous similar messages [ 2887.708280] Lustre: Skipped 2 previous similar messages [ 2892.755900] Lustre: server umount lustre-MDT0000 complete [ 2892.778860] LustreError: 96864:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2892.795086] LustreError: 96864:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2896.487722] LustreError: 93968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789581450 with bad export cookie 9414348917519352921 [ 2896.491460] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2896.510469] LustreError: 93968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2897.226289] Lustre: server umount lustre-MDT0001 complete [ 2911.812129] Lustre: server umount lustre-OST0000 complete [ 2929.172123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2930.276145] LustreError: 98299:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2930.652729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2935.782879] LustreError: 98300:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2937.759455] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2946.017173] LustreError: 98299:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2946.025396] LustreError: 98299:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2947.486879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2948.449108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2953.840946] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2963.952551] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2970.984532] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2985.881449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2986.159647] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2986.172164] LustreError: 98287:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2986.195977] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2986.201882] LustreError: 100384:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2986.206915] LustreError: 100384:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2986.211140] LustreError: 100384:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2986.293863] Lustre: server umount lustre-OST0000 complete [ 2986.295672] LustreError: 100384:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3001.013424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3001.239508] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3001.253297] LustreError: 98288:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3001.288659] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3001.302403] LustreError: 100850:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 3001.312474] LustreError: 100850:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 3001.328131] LustreError: 100850:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3001.441453] Lustre: server umount lustre-OST0000 complete [ 3001.445340] LustreError: 100850:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3014.837586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3014.975709] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3014.978707] Lustre: Found index 0 for lustre-OST0000, updating log [ 3014.982389] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3015.100638] Lustre: lustre-OST0000: new disk, initializing [ 3015.119040] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3015.133165] Lustre: 101386:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3015.469070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3023.451923] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3028.381765] Lustre: Failing over lustre-OST0000 [ 3028.592981] Lustre: server umount lustre-OST0000 complete [ 3035.113516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3035.120552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3035.139281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3037.167625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3037.180593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3042.277749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3042.286707] Lustre: Skipped 2 previous similar messages [ 3043.437606] Lustre: server umount lustre-MDT0000 complete [ 3045.344059] Lustre: 93296:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789581583/real 1789581583] req@00000000a5a335f4 x1876511987708672/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789581599 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3045.397026] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3045.427086] LustreError: 99040:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3045.456669] LustreError: 99040:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3047.376359] LustreError: 98281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789581601 with bad export cookie 9414348917519355665 [ 3047.377101] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3047.404356] LustreError: 98281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3047.713829] Lustre: server umount lustre-MDT0001 complete [ 3059.824659] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 14:00:10 (1789581610) [ 3080.680444] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 3093.015404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3093.924249] LustreError: 103733:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3094.087551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3101.845041] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3117.609599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3128.587765] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3132.857572] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3142.312652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3143.113662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3143.118379] Lustre: Skipped 1 previous similar message [ 3143.149750] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3145.136230] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3150.811266] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3160.978513] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3161.057436] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3161.133362] Lustre: lustre-OST0001: new disk, initializing [ 3161.138089] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3161.143293] Lustre: 106068:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3164.655650] Lustre: lustre-OST0000: Recovery over after 0:19, of 2 clients 2 recovered and 0 were evicted. [ 3164.659411] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3164.679541] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 3168.075452] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3168.428128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3168.435055] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3168.529488] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3176.472520] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3184.272245] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3191.909270] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.48@tcp (stopping) [ 3192.802768] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3192.812855] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3196.372856] Lustre: server umount lustre-MDT0001 complete [ 3196.979029] LustreError: 104643:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3197.022678] LustreError: 104643:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 3207.968730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3208.088965] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3208.091350] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3208.155818] Lustre: 103722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3208.822222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3208.826111] Lustre: Skipped 1 previous similar message [ 3209.864356] LustreError: 107642:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3209.875460] LustreError: 107642:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3209.889943] LustreError: 103726:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3210.030916] LustreError: 107654:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.148@tcp: cfg command failed: rc = -17 [ 3210.036678] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3210.036678] [ 3213.689972] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3214.332828] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3214.346284] Lustre: Skipped 1 previous similar message [ 3225.811332] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 14:02:57 (1789581777) [ 3249.042664] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 3268.821505] Lustre: 109709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3294.466295] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3298.816977] Lustre: 110845:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3392.997913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3393.028506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3393.040249] Lustre: Skipped 2 previous similar messages [ 3393.050685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3393.055674] Lustre: Skipped 4 previous similar messages [ 3395.117185] Lustre: server umount lustre-MDT0000 complete [ 3398.624471] LustreError: 103729:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3398.645468] LustreError: 103729:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 3400.388539] LustreError: 104488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789581954 with bad export cookie 9414348917519356673 [ 3400.398380] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3401.092720] Lustre: server umount lustre-MDT0001 complete [ 3416.163208] Lustre: server umount lustre-OST0000 complete [ 3419.615683] Lustre: 93298:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789581957/real 1789581957] req@ffff8a35fb6bfb80 x1876511987979136/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789581973 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3419.643970] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3419.668657] Lustre: Skipped 3 previous similar messages [ 3421.157862] Lustre: 93295:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789581959/real 1789581959] req@ffff8a34c21a8e00 x1876511987979392/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789581975 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3422.973650] Lustre: server umount lustre-OST0001 complete [ 3432.459273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3445.643434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3455.961888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3467.393068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3488.208114] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 3499.728453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3499.778585] Lustre: lustre-MDT0000: reset Object Index mappings [ 3500.514259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3506.691644] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3516.019938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3516.042447] Lustre: lustre-MDT0001: reset Object Index mappings [ 3523.282741] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3526.990379] Lustre: 115948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3535.408662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3541.374394] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:227 to 0x280000400:257) [ 3544.022267] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3545.463209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3553.301887] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3559.427223] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3559.444930] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3560.666411] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3568.893843] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3574.078347] Lustre: 117825:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3620.486107] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 14:09:31 (1789582171) [ 3649.505181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3649.515125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3649.530651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3649.536118] Lustre: Skipped 4 previous similar messages [ 3652.961220] Lustre: server umount lustre-MDT0000 complete [ 3656.682830] LustreError: 115739:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3656.707068] LustreError: 114778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789582210 with bad export cookie 9414348917519403727 [ 3656.708809] LustreError: 115739:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 3656.710338] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3656.765642] LustreError: 114778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3663.278749] Lustre: server umount lustre-MDT0001 complete [ 3673.372985] Lustre: server umount lustre-OST0000 complete [ 3684.041258] Lustre: server umount lustre-OST0001 complete [ 3691.138882] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 3702.067764] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 3743.006348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3743.195635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3743.225680] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3743.322128] Lustre: lustre-MDT0000: new disk, initializing [ 3743.423983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3747.943255] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3755.757133] Lustre: server umount lustre-MDT0000 complete [ 3767.466882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3768.020096] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3768.032983] Lustre: 122091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3768.047683] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3768.385158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3768.389152] Lustre: Skipped 4 previous similar messages [ 3773.689530] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3779.391658] Lustre: server umount lustre-MDT0000 complete [ 3789.257974] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 14:12:20 (1789582340) [ 3791.509892] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3794.197298] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 14:12:25 (1789582345) [ 3815.386726] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 3826.325340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3831.693525] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3842.103690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3842.228586] Lustre: 124144:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3842.350576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3842.367643] Lustre: Skipped 1 previous similar message [ 3842.519685] Lustre: lustre-MDT0001: new disk, initializing [ 3842.983118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3842.999159] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3848.534448] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3855.341618] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3864.372516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3864.772868] Lustre: lustre-OST0000: new disk, initializing [ 3864.776731] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3864.784292] Lustre: 125733:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3866.203618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3866.221782] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3866.448838] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3872.528497] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3884.402346] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3884.594964] Lustre: lustre-OST0001: new disk, initializing [ 3884.598943] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3884.609699] Lustre: 126604:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3886.250173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3886.262705] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3886.391493] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3891.540980] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3902.507220] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3906.872927] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4353.486955] Lustre: 133029:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4353.492445] Lustre: 133029:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4353.496743] Lustre: 133029:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4353.500945] Lustre: 133029:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4353.506985] Lustre: 133029:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4353.511256] Lustre: 133029:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4655.547119] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:26:46 (1789583206) [ 4670.947996] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4670.954112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4670.962535] Lustre: Skipped 7 previous similar messages [ 4670.985820] Lustre: Skipped 9 previous similar messages [ 4676.070850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4676.076504] Lustre: Skipped 3 previous similar messages [ 4685.281400] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4685.821849] Lustre: server umount lustre-MDT0000 complete [ 4686.304567] LustreError: 124154:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4686.321230] LustreError: 124154:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4690.946256] LustreError: 127425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789583245 with bad export cookie 9414348917519470269 [ 4690.948234] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4690.959821] LustreError: 127425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4691.436911] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4691.473071] Lustre: Skipped 1 previous similar message [ 4691.484785] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4691.493754] Lustre: Skipped 5 previous similar messages [ 4697.518476] Lustre: server umount lustre-MDT0001 complete [ 4707.855547] Lustre: server umount lustre-OST0000 complete [ 4717.769661] Lustre: server umount lustre-OST0001 complete [ 4724.891227] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 4738.828940] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 4777.926459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4778.125697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4778.144194] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4778.204289] Lustre: lustre-MDT0000: new disk, initializing [ 4778.253264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4778.256399] Lustre: Skipped 4 previous similar messages [ 4778.266843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4782.948809] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4793.594089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4793.849069] Lustre: 136088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4793.956734] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4793.963394] Lustre: Skipped 1 previous similar message [ 4794.106083] Lustre: lustre-MDT0001: new disk, initializing [ 4794.328934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4794.343856] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4799.339851] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4809.450691] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4816.093346] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4823.258604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4823.457657] Lustre: lustre-OST0000: new disk, initializing [ 4823.460532] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4823.464025] Lustre: 138055:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4825.196564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4825.213455] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4825.495187] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4831.236281] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4841.014287] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4850.474252] Lustre: lustre-MDD0000: changelog on [ 4853.087296] Lustre: lustre-MDD0001: changelog on [ 6825.412996] Lustre: lustre-MDD0001: changelog off [ 6828.084063] Lustre: 139868:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6829.411956] Lustre: lustre-MDD0000: changelog off [ 6837.732969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6837.760952] Lustre: Skipped 1 previous similar message [ 6837.781838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6837.791324] Lustre: Skipped 3 previous similar messages [ 6842.853570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6842.865679] Lustre: Skipped 1 previous similar message [ 6847.972687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6848.002251] Lustre: Skipped 1 previous similar message [ 6851.039318] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6851.249920] Lustre: server umount lustre-OST0000 complete [ 6857.703337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6857.712112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6857.730888] Lustre: Skipped 1 previous similar message [ 6857.738426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6861.980378] Lustre: server umount lustre-MDT0000 complete [ 6863.339344] LustreError: 136895:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6863.367932] LustreError: 136895:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6866.813847] LustreError: 138054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789585420 with bad export cookie 9414348917520292783 [ 6866.817297] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6866.829821] LustreError: 138054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6867.361932] Lustre: server umount lustre-MDT0001 complete [ 6880.573929] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 6884.000214] Key type lgssc unregistered [ 6884.353398] LNet: 147223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6884.360519] LNetError: 147223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6884.379429] LNet: Removed LNI 192.168.201.148@tcp [ 6885.318301] Key type .llcrypt unregistered [ 6885.320096] Key type ._llcrypt unregistered [ 6910.752799] Key type ._llcrypt registered [ 6910.754333] Key type .llcrypt registered [ 6910.929889] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 6927.740742] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 6929.032256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6929.182188] alg: No test for adler32 (adler32-zlib) [ 6930.247755] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 6930.498538] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 6932.255442] Key type lgssc registered [ 6933.868625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6979.130182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6979.187919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6980.549204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6980.583349] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6980.654467] Lustre: lustre-MDT0000: new disk, initializing [ 6980.740882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6980.758211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6985.721180] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6998.045309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6998.248266] Lustre: 150421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6998.310229] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6998.316577] Lustre: Skipped 1 previous similar message [ 6998.451558] Lustre: lustre-MDT0001: new disk, initializing [ 6998.607374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6998.643881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6998.657630] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7004.615262] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7014.694216] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7020.136211] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7028.113506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7028.496090] Lustre: lustre-OST0000: new disk, initializing [ 7028.502440] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7028.516348] Lustre: 152389:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7028.627615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7034.933236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7034.938525] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7035.042358] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7035.707691] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7045.508819] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7052.392870] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_module obdecho/obdecho [ 7054.204151] LustreError: 153392:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7054.216821] LustreError: 153392:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7056.488810] LustreError: 153488:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7070.178280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7070.188794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7070.217805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7071.208495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7071.209402] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7074.467792] Lustre: server umount lustre-MDT0000 complete [ 7079.086726] LustreError: 150414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789585633 with bad export cookie 9442550775313239677 [ 7079.094386] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7079.106762] LustreError: 150414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7079.497258] Lustre: server umount lustre-MDT0001 complete [ 7093.886206] Lustre: server umount lustre-OST0000 complete [ 7121.898985] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 7133.732824] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 7172.008358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7172.313054] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7172.370773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7172.508535] Lustre: lustre-MDT0000: new disk, initializing [ 7172.640396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7172.663865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7177.385806] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7189.491494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7189.680065] Lustre: 157209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7189.742524] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7189.746743] Lustre: Skipped 1 previous similar message [ 7189.917430] Lustre: lustre-MDT0001: new disk, initializing [ 7190.099219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7190.138360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7190.157881] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7195.629455] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7206.116833] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7212.083686] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7220.579714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7220.933331] Lustre: lustre-OST0000: new disk, initializing [ 7220.936942] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7220.944239] Lustre: 159178:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7221.034307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7227.334505] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7229.020937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7229.038346] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7229.253674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7237.826147] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7242.953964] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7243.164557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7248.005124] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7248.249897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7255.014501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7255.024110] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7255.031701] Lustre: Skipped 1 previous similar message [ 7255.035196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7255.040317] LustreError: Skipped 1 previous similar message [ 7255.053841] Lustre: Skipped 3 previous similar messages [ 7256.470724] Lustre: server umount lustre-OST0000 complete [ 7261.664981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7261.667328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7261.673159] Lustre: Skipped 1 previous similar message [ 7261.678818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7267.139465] Lustre: server umount lustre-MDT0000 complete [ 7270.371081] LustreError: 157216:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7270.385936] LustreError: 157216:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 7270.606431] LustreError: 159177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789585824 with bad export cookie 9442550775313240636 [ 7270.620503] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7270.622073] LustreError: 159177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7270.913276] Lustre: server umount lustre-MDT0001 complete [ 7274.263832] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 15:10:25 (1789585825) [ 7275.977570] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7278.176359] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 15:10:29 (1789585829) [ 7286.332359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7287.101721] LustreError: 161590:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7287.238710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7292.002996] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7292.388151] LustreError: 161591:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7297.507536] LustreError: 161590:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7302.346094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7309.596143] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7319.008847] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7325.915927] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7333.512586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7333.952508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7333.960976] Lustre: Skipped 1 previous similar message [ 7341.124303] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7349.452818] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7367.649161] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7367.657907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7367.670921] Lustre: Skipped 2 previous similar messages [ 7367.692765] Lustre: Skipped 4 previous similar messages [ 7372.157612] Lustre: server umount lustre-MDT0000 complete [ 7372.770364] LustreError: 162318:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7372.792861] LustreError: 162318:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7376.302687] LustreError: 161570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789585930 with bad export cookie 9442550775313241518 [ 7376.309185] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7376.326914] LustreError: 161570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7377.895590] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7377.912728] Lustre: Skipped 1 previous similar message [ 7377.919591] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7377.923398] Lustre: Skipped 1 previous similar message [ 7381.045213] LustreError: 161586:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7381.063225] LustreError: 161586:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7382.656717] Lustre: server umount lustre-MDT0001 complete [ 7389.470868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7404.982651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7405.234897] LustreError: 165456:0:(llog.c:1655:llog_backup()) MGC192.168.201.148@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7405.239207] Lustre: 165456:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.148@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7406.561358] LustreError: 165461:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7406.591302] LustreError: 165461:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7441.376923] LustreError: 165462:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7441.395424] LustreError: 165462:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 26 previous similar messages [ 7466.463394] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7466.471881] Lustre: 165456:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.148@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7507.451513] LustreError: 165480:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7507.452763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7507.484556] LustreError: 165480:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 42 previous similar messages [ 7512.551240] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7513.894727] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7524.062911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7538.152871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7547.378239] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x830aade6c022af07 [ 7547.404705] Lustre: MGC192.168.201.148@tcp: Connection restored to 0@lo (at 0@lo) [ 7552.507870] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7553.007737] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7559.890677] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7561.602375] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7567.535697] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7569.206833] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7593.970433] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7593.975654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7593.999177] Lustre: Skipped 1 previous similar message [ 7594.009520] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7594.020657] Lustre: Skipped 4 previous similar messages [ 7596.621500] Lustre: server umount lustre-MDT0000 complete [ 7600.914053] LustreError: 161571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789586155 with bad export cookie 9442550775313248007 [ 7600.937546] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7601.547665] Lustre: server umount lustre-MDT0001 complete [ 7616.561008] Lustre: server umount lustre-OST0000 complete [ 7641.673764] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 7653.674721] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 7697.545964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7697.774301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7697.793387] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7697.883770] Lustre: lustre-MDT0000: new disk, initializing [ 7697.947379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7697.953661] Lustre: Skipped 1 previous similar message [ 7697.969977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7702.472691] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7713.045234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7713.165313] Lustre: 170568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7713.448726] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7718.577888] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7729.941593] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7739.240630] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7747.292149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7747.654167] Lustre: lustre-OST0000: new disk, initializing [ 7747.658048] Lustre: Skipped 1 previous similar message [ 7747.664294] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7747.669402] Lustre: Skipped 2 previous similar messages [ 7747.675219] Lustre: 172538:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7755.881069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7755.893897] Lustre: Skipped 1 previous similar message [ 7755.908380] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7756.085688] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7756.313775] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7768.552838] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7774.551667] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7774.844830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7779.927148] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7780.290653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7786.983180] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7786.992491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7787.004107] Lustre: Skipped 1 previous similar message [ 7787.012134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7788.742294] Lustre: server umount lustre-OST0000 complete [ 7799.583747] Lustre: server umount lustre-MDT0000 complete [ 7802.337155] LustreError: 171743:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7802.373970] LustreError: 171743:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 36 previous similar messages [ 7803.761907] LustreError: 170561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789586357 with bad export cookie 9442550775313264807 [ 7803.762717] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7803.768352] LustreError: 170561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7804.378225] Lustre: server umount lustre-MDT0001 complete [ 7815.106191] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 15:19:26 (1789586366) [ 7823.122721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7829.631693] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7842.066091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7848.221621] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7858.665347] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7866.887815] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7875.277021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7883.905601] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7893.158567] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7910.644525] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 15:21:01 (1789586461) [ 7918.049773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7918.069281] LustreError: Skipped 2 previous similar messages [ 7918.080706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7918.098238] Lustre: Skipped 3 previous similar messages [ 7918.110977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7918.113498] Lustre: Skipped 4 previous similar messages [ 7922.121906] Lustre: server umount lustre-OST0000 complete [ 7933.195355] Lustre: server umount lustre-MDT0000 complete [ 7937.595947] LustreError: 174930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789586491 with bad export cookie 9442550775313265689 [ 7937.618925] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7937.624129] LustreError: 174930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7938.127242] Lustre: server umount lustre-MDT0001 complete [ 7951.381363] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 7954.320450] Key type lgssc unregistered [ 7954.625599] LNet: 178681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7954.633505] LNetError: 178681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7954.676527] LNet: Removed LNI 192.168.201.148@tcp [ 7955.710250] Key type .llcrypt unregistered [ 7955.712667] Key type ._llcrypt unregistered [ 7976.220345] Key type ._llcrypt registered [ 7976.221507] Key type .llcrypt registered [ 7976.372512] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 7978.162703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7978.191544] alg: No test for adler32 (adler32-zlib) [ 7979.437773] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 7979.789694] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 7981.511379] Key type lgssc registered [ 7983.150366] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7992.993515] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7993.006413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8008.607914] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8013.727427] LustreError: 179922:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.148@tcp: failed processing log, type 4: rc = -110 [ 8018.911475] LustreError: 179922:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8018.927899] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8018.930657] LustreError: MGC192.168.201.148@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8018.953948] LustreError: 179922:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8018.968392] LustreError: 179922:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 8018.980866] LustreError: 179922:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 8019.090646] Lustre: server umount lustre-OST0000 complete [ 8019.095449] LustreError: 179922:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8029.361484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8031.009448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8037.164129] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8041.451676] LustreError: 180421:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8041.473749] LustreError: 180421:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8046.577378] LustreError: 180420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8048.418453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8049.110663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8054.446524] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8064.531648] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8072.257664] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8080.802940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8081.122544] LustreError: 182239:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8081.156801] Lustre: 182239:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.148@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8081.314667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8089.171916] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8098.090341] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8101.071854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8101.088443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8101.088681] Lustre: Skipped 2 previous similar messages [ 8101.108819] Lustre: Skipped 2 previous similar messages [ 8106.465675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8106.477170] Lustre: Skipped 2 previous similar messages [ 8107.581336] Lustre: server umount lustre-MDT0000 complete [ 8111.589896] LustreError: 180417:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8111.619968] LustreError: 180417:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8111.721403] LustreError: 180402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789586665 with bad export cookie 10341661945160109205 [ 8111.728981] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8111.733222] LustreError: 180402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8112.220609] Lustre: server umount lustre-MDT0001 complete [ 8127.728701] Lustre: server umount lustre-OST0000 complete [ 8140.049635] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 8155.405708] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 8201.109343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8201.611542] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8201.691211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8201.859426] Lustre: lustre-MDT0000: new disk, initializing [ 8201.960196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8201.975555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8207.643094] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8220.146200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8220.313329] Lustre: 185784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8220.392323] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8220.395700] Lustre: Skipped 1 previous similar message [ 8220.498886] Lustre: lustre-MDT0001: new disk, initializing [ 8220.660149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8220.696432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8220.706926] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8226.054586] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8237.181576] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8242.992891] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8250.793534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8251.034253] Lustre: lustre-OST0000: new disk, initializing [ 8251.038588] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8251.047166] Lustre: 187750:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8251.151346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8258.641965] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8260.150922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8260.160991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8260.289241] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8269.127222] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8275.003200] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8275.323633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8280.912299] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8281.176826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8286.183166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8286.188397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8286.226665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8286.233889] Lustre: Skipped 1 previous similar message [ 8289.681436] Lustre: server umount lustre-OST0000 complete [ 8296.416350] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8296.424763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8296.434477] Lustre: Skipped 2 previous similar messages [ 8296.456805] Lustre: Skipped 1 previous similar message [ 8300.736181] Lustre: server umount lustre-MDT0000 complete [ 8301.537376] LustreError: 185793:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8301.552471] LustreError: 185793:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8304.767035] LustreError: 185776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789586858 with bad export cookie 10341661945160109912 [ 8304.770677] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8304.776306] LustreError: 185776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8305.179038] Lustre: server umount lustre-MDT0001 complete [ 8315.805639] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 15:27:46 (1789586866) [ 8340.982265] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 8344.180818] Key type lgssc unregistered [ 8344.568038] LNet: 190697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8344.580420] LNetError: 190697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8344.598763] LNet: Removed LNI 192.168.201.148@tcp [ 8345.797756] Key type .llcrypt unregistered [ 8345.804066] Key type ._llcrypt unregistered [ 8367.511972] Key type ._llcrypt registered [ 8367.517079] Key type .llcrypt registered [ 8367.669833] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 8369.366736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8369.775246] alg: No test for adler32 (adler32-zlib) [ 8370.862883] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 8371.292693] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 8372.976169] Key type lgssc registered [ 8374.499287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8383.833793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8383.851609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8385.394754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8390.653140] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8395.752067] LustreError: 191914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8395.778647] LustreError: 191914:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8400.864658] LustreError: 191913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8401.125851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8401.529497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8406.006685] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8414.382912] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8421.892461] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8431.275802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8431.575382] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8431.579457] LustreError: 193776:0:(llog.c:1696:llog_backup()) MGC192.168.201.148@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8431.590500] Lustre: 193776:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.148@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8431.739951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8438.904533] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8447.914342] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8451.563814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8451.572698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8451.583032] Lustre: Skipped 2 previous similar messages [ 8451.592177] Lustre: Skipped 2 previous similar messages [ 8456.664194] Lustre: server umount lustre-MDT0000 complete [ 8456.691506] LustreError: 191913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8456.719952] LustreError: 191913:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8461.241699] LustreError: 191894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789587015 with bad export cookie 15458316618269042575 [ 8461.246223] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8461.260616] LustreError: 191894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8461.615834] Lustre: server umount lustre-MDT0001 complete [ 8476.875327] Lustre: server umount lustre-OST0000 complete [ 8489.714500] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 8501.664963] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 8549.114341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8549.449439] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8549.552532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8549.657400] Lustre: lustre-MDT0000: new disk, initializing [ 8549.753320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8549.766276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8555.071463] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8567.495699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8567.691469] Lustre: 197308:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8567.788674] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8567.794241] Lustre: Skipped 1 previous similar message [ 8567.798918] LustreError: 198091:0:(llog.c:1696:llog_backup()) MGC192.168.201.148@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8567.807587] LustreError: 198091:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8567.817105] Lustre: 198091:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.148@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8567.867398] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8567.872809] Lustre: Skipped 1 previous similar message [ 8568.031764] Lustre: lustre-MDT0001: new disk, initializing [ 8568.282641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8568.335469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8568.354326] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8573.896229] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8585.832800] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8591.947802] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8598.921050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8599.174673] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8599.177571] Lustre: Skipped 1 previous similar message [ 8599.182691] LustreError: 199254:0:(llog.c:1696:llog_backup()) MGC192.168.201.148@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8599.196313] LustreError: 199254:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8599.202268] Lustre: 199254:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.148@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8599.211312] Lustre: 199254:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8599.230428] Lustre: lustre-OST0000: new disk, initializing [ 8599.233436] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8599.237878] Lustre: 199277:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8599.311229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8605.827949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8605.845873] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8606.021812] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8606.054306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8615.959816] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8621.905368] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8622.210880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8627.878116] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8628.322067] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8631.799175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8631.817991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8631.829421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8636.635805] Lustre: server umount lustre-OST0000 complete [ 8642.019957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8642.037792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8642.037869] Lustre: Skipped 2 previous similar messages [ 8642.059379] Lustre: Skipped 2 previous similar messages [ 8647.138582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8647.146169] Lustre: Skipped 1 previous similar message [ 8647.711227] Lustre: server umount lustre-MDT0000 complete [ 8651.553383] LustreError: 197299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789587205 with bad export cookie 15458316618269043282 [ 8651.554026] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8651.559076] LustreError: 197299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8652.033612] Lustre: server umount lustre-MDT0001 complete [ 8662.481171] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 15:33:33 (1789587213) [ 8684.864470] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 8695.711926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8696.011364] LustreError: 202407:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8696.061187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8700.981352] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8701.435589] LustreError: 202408:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8710.560978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8715.936208] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8718.917318] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8725.563647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8733.082422] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8743.382502] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8743.648676] Lustre: lustre-OST0001: new disk, initializing [ 8743.652189] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8743.660098] Lustre: 204713:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8743.778855] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8743.796962] Lustre: Skipped 2 previous similar messages [ 8750.587564] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8752.728932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8752.744052] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8752.864294] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8764.172776] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8776.835718] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8779.020468] Lustre: DEBUG MARKER: ADD OST3 [ 8783.805886] loop: module loaded [ 8783.920503] loop0: detected capacity change from 0 to 204800000 [ 8784.328954] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8784.349242] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8784.361783] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8791.103636] loop0: detected capacity change from 0 to 204800000 [ 8792.014864] Lustre: DEBUG MARKER: STOP OST3 [ 8797.671412] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8797.887769] Lustre: lustre-OST0003: new disk, initializing [ 8797.891232] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8797.896381] Lustre: 206712:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8799.523433] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8799.674872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8799.687904] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8799.734042] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8801.575267] LustreError: 206872:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8802.277085] Lustre: Failing over lustre-OST0003 [ 8802.500441] Lustre: server umount lustre-OST0003 complete [ 8804.836037] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8804.851524] LustreError: Skipped 1 previous similar message [ 8804.853611] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8804.863805] LustreError: 203891:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8804.876624] LustreError: 203891:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8806.276411] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8816.618927] Lustre: DEBUG MARKER: START OST3 again [ 8824.177851] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8824.443908] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8824.448155] Lustre: Skipped 1 previous similar message [ 8824.471255] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8825.395392] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8826.220807] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8826.245537] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8831.115793] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8853.050352] Lustre: Failing over lustre-OST0003 [ 8853.184184] Lustre: server umount lustre-OST0003 complete [ 8855.541476] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8855.549024] LustreError: 203893:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8855.558415] Lustre: Skipped 2 previous similar messages [ 8855.589630] LustreError: 203893:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 8874.978618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8874.992915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8875.021378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8884.201915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8884.222515] Lustre: Skipped 11 previous similar messages [ 8888.287136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8888.522431] Lustre: server umount lustre-MDT0000 complete [ 8889.319081] LustreError: 202402:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8889.348750] LustreError: 202402:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 8892.875644] LustreError: 202389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789587446 with bad export cookie 15458316618269044171 [ 8892.899411] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8894.436545] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8894.452023] Lustre: Skipped 3 previous similar messages [ 8902.647276] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8902.659925] Lustre: Skipped 5 previous similar messages [ 8907.231124] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8907.532180] Lustre: server umount lustre-MDT0001 complete [ 8911.399228] Lustre: server umount lustre-OST0000 complete [ 8915.070235] Lustre: server umount lustre-OST0001 complete [ 8921.794673] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 8931.821656] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 8972.247106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8972.573503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8972.615228] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8972.747896] Lustre: lustre-MDT0000: new disk, initializing [ 8972.823157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8972.837811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8977.845818] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8989.568528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8989.719857] Lustre: 212180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8989.773882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8989.777818] Lustre: Skipped 1 previous similar message [ 8989.859650] Lustre: lustre-MDT0001: new disk, initializing [ 8990.079743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8990.090437] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8994.546306] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9005.100648] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9011.376534] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9019.128649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9019.584543] Lustre: 214150:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9021.481730] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9021.661654] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9027.371449] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9036.321839] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9041.410638] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9041.666120] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9047.761227] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9047.958269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9052.640528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9052.644511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9052.655032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9052.655042] Lustre: Skipped 2 previous similar messages [ 9052.688176] Lustre: Skipped 2 previous similar messages [ 9057.249857] Lustre: server umount lustre-OST0000 complete [ 9077.218069] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9077.465953] Lustre: server umount lustre-MDT0000 complete [ 9082.123938] LustreError: 212171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789587636 with bad export cookie 15458316618269046467 [ 9082.133342] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9082.139680] LustreError: 212171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9082.593159] Lustre: server umount lustre-MDT0001 complete [ 9087.475113] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 15:40:38 (1789587638) [ 9104.499484] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 9122.089941] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 9168.695476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9169.015930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9169.049161] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9169.053918] Lustre: Skipped 1 previous similar message [ 9169.188914] Lustre: lustre-MDT0000: new disk, initializing [ 9169.191972] Lustre: Skipped 1 previous similar message [ 9169.322953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9169.327908] Lustre: Skipped 1 previous similar message [ 9174.787872] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9186.100457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9186.323566] Lustre: 218786:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9186.681300] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9192.370718] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9203.948612] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9210.935596] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9220.101126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9220.496109] Lustre: 220754:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9221.624549] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9221.766525] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9228.333511] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9240.160298] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9246.149097] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9246.380576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9251.205693] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9251.479968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9257.451091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9257.454025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9257.472338] LustreError: Skipped 1 previous similar message [ 9257.478394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9257.498434] Lustre: Skipped 3 previous similar messages [ 9257.513138] Lustre: Skipped 9 previous similar messages [ 9259.723442] Lustre: server umount lustre-OST0000 complete [ 9272.250186] Lustre: server umount lustre-MDT0000 complete [ 9272.807487] LustreError: 219664:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9272.827367] LustreError: 219664:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 9276.581793] LustreError: 218779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789587830 with bad export cookie 15458316618269047349 [ 9276.593235] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9276.597634] LustreError: 218779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9276.992733] Lustre: server umount lustre-MDT0001 complete [ 9288.082987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9288.756875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9288.767660] Lustre: Skipped 5 previous similar messages [ 9295.492707] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9305.359965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9311.620500] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9321.457679] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9329.138364] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9337.231674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9344.479468] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9353.339795] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9369.570559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9375.620557] Lustre: server umount lustre-OST0000 complete [ 9388.020433] Lustre: server umount lustre-MDT0000 complete [ 9392.024368] LustreError: 222955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789587946 with bad export cookie 15458316618269048231 [ 9392.026578] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9392.037546] LustreError: 222955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9392.535533] Lustre: server umount lustre-MDT0001 complete [ 9496.155533] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 9499.360745] Key type lgssc unregistered [ 9499.720578] LNet: 226660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9499.730574] LNetError: 226660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9499.746899] LNet: Removed LNI 192.168.201.148@tcp [ 9500.763390] Key type .llcrypt unregistered [ 9500.766327] Key type ._llcrypt unregistered [ 9529.451655] Key type ._llcrypt registered [ 9529.454867] Key type .llcrypt registered [ 9529.585664] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 9548.514488] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 9550.327274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9550.402201] alg: No test for adler32 (adler32-zlib) [ 9551.454696] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 9551.645917] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 9553.279210] Key type lgssc registered [ 9554.786297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9604.057199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9604.122983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9605.355130] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9605.389563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9605.487762] Lustre: lustre-MDT0000: new disk, initializing [ 9605.601509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9605.621982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9610.935548] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9622.985053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9623.106815] Lustre: 230013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9623.162525] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9623.170422] Lustre: Skipped 1 previous similar message [ 9623.235900] Lustre: lustre-MDT0001: new disk, initializing [ 9623.332738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9623.350939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9623.363182] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9628.503577] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9640.885440] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9648.652439] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9656.208989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9656.401114] Lustre: lustre-OST0000: new disk, initializing [ 9656.407327] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9656.414266] Lustre: 231986:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9656.556711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9664.091910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9664.106105] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9664.284704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9664.417930] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9675.015803] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9681.189892] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9681.530486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9687.326260] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9687.821964] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9695.218294] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9695.227966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9695.250255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9696.594603] Lustre: server umount lustre-OST0000 complete [ 9705.443446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9705.445229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9705.446783] LustreError: Skipped 1 previous similar message [ 9705.459273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9705.486495] Lustre: Skipped 2 previous similar messages [ 9705.492286] Lustre: Skipped 1 previous similar message [ 9707.566043] Lustre: server umount lustre-MDT0000 complete [ 9711.762969] LustreError: 230007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789588265 with bad export cookie 8843041229995279217 [ 9711.773895] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9711.775556] LustreError: 230007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9712.067301] Lustre: server umount lustre-MDT0001 complete [ 9734.741729] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 9747.192339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9747.819327] LustreError: 234919:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9747.844296] LustreError: 234919:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9747.981637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9753.068179] LustreError: 234920:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9754.725809] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9757.160706] LustreError: 234919:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9762.281105] LustreError: 234920:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9763.984983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9764.608989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9770.226695] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9773.517808] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9781.172567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9781.746993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9789.904140] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9799.745025] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9800.008562] Lustre: lustre-OST0001: new disk, initializing [ 9800.010787] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9800.014442] Lustre: 237228:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9800.084275] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9806.456758] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9809.542474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9809.552489] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9809.902190] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9822.358128] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9832.339987] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9846.484735] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 15:53:18 (1789588398) [ 9850.849651] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9850.857393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9850.870615] Lustre: Skipped 2 previous similar messages [ 9850.887720] Lustre: Skipped 4 previous similar messages [ 9855.567713] Lustre: server umount lustre-MDT0000 complete [ 9855.968692] LustreError: 234916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9855.987799] LustreError: 234916:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9860.108121] LustreError: 234900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789588414 with bad export cookie 8843041229995280099 [ 9860.112105] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9860.126616] LustreError: 234900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9860.368873] Lustre: server umount lustre-MDT0001 complete [ 9874.396491] Lustre: server umount lustre-OST0000 complete [ 9877.228145] Lustre: 228235:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789588415/real 1789588415] req@ffff8a34c6a61880 x1876519136907392/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789588431 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9877.261556] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9877.267359] Lustre: Skipped 1 previous similar message [ 9879.340203] Lustre: server umount lustre-OST0001 complete [ 9887.337929] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 9898.557633] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 9945.609683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9945.777885] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9945.801473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9945.912200] Lustre: lustre-MDT0000: new disk, initializing [ 9946.022549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9946.087201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9951.922467] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9965.843960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9966.005615] Lustre: 241180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9966.078120] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9966.084971] Lustre: Skipped 1 previous similar message [ 9966.163392] Lustre: lustre-MDT0001: new disk, initializing [ 9966.391982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9966.406713] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9973.825199] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9985.151889] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9991.122884] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9998.814185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9999.169521] Lustre: lustre-OST0000: new disk, initializing [ 9999.172725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9999.181882] Lustre: 243149:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10000.867302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10000.884989] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10001.031848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10006.437751] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10016.633088] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10022.409322] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10022.692030] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10027.599895] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10027.740739] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10031.592826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10031.594525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10031.615168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10031.624584] LustreError: Skipped 1 previous similar message [10031.627290] Lustre: Skipped 1 previous similar message [10036.037791] Lustre: server umount lustre-OST0000 complete [10041.840144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10041.852162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10041.863908] Lustre: Skipped 2 previous similar messages [10041.895819] Lustre: Skipped 1 previous similar message [10046.946789] LustreError: 243241:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10047.133921] Lustre: server umount lustre-MDT0000 complete [10052.204348] LustreError: 241172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789588606 with bad export cookie 8843041229995282234 [10052.205745] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10052.213497] LustreError: 241172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10052.632749] Lustre: server umount lustre-MDT0001 complete [10061.507043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10061.924820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10061.927265] Lustre: Skipped 2 previous similar messages [10066.916950] LustreError: 245367:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10066.940494] LustreError: 245367:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10066.954155] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10078.616741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10084.735354] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10094.898213] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10101.720410] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10109.703362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10119.305924] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10129.266053] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10277.856542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10277.864679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10277.879073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10282.477934] Lustre: server umount lustre-OST0000 complete [10289.120431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10289.129247] LustreError: Skipped 1 previous similar message [10293.378408] Lustre: server umount lustre-MDT0000 complete [10295.781504] LustreError: 245367:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10295.809492] LustreError: 245367:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10297.052231] LustreError: 245348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789588851 with bad export cookie 8843041229995283116 [10297.053784] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10297.059568] LustreError: 245348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10297.370754] Lustre: server umount lustre-MDT0001 complete [10402.279759] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [10405.388264] Key type lgssc unregistered [10405.700568] LNet: 248988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10405.712052] LNetError: 248988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10405.737347] LNet: Removed LNI 192.168.201.148@tcp [10406.804156] Key type .llcrypt unregistered [10406.805871] Key type ._llcrypt unregistered [10437.325339] Key type ._llcrypt registered [10437.327615] Key type .llcrypt registered [10437.481582] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [10453.873692] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [10455.674559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10455.684270] alg: No test for adler32 (adler32-zlib) [10456.886584] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [10457.138850] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [10458.823213] Key type lgssc registered [10460.394619] Lustre: Echo OBD driver; http://www.lustre.org/ [10515.228273] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [10526.738508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10526.777236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10528.189265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10528.239881] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10528.334752] Lustre: lustre-MDT0000: new disk, initializing [10528.462537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10528.501155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10534.491588] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10546.175818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10546.364407] Lustre: 253072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10546.445810] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10546.451118] Lustre: Skipped 1 previous similar message [10546.527408] Lustre: lustre-MDT0001: new disk, initializing [10546.641613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10546.712869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10546.729636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10551.834226] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10556.633660] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10563.112268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10563.451345] Lustre: lustre-OST0000: new disk, initializing [10563.457071] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10563.466900] Lustre: 254706:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10563.598724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10570.312343] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10572.361954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10572.372442] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10572.509689] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10580.406329] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10580.542576] Lustre: lustre-OST0001: new disk, initializing [10580.546289] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10580.551806] Lustre: 255578:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10580.675082] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10586.767911] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10588.824854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10588.837673] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10588.994810] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10600.696599] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10610.562793] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10628.578815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10628.594818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10628.620910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10630.114295] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10630.118314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10630.233339] Lustre: Skipped 3 previous similar messages [10632.596517] Lustre: server umount lustre-MDT0000 complete [10637.313790] LustreError: 254705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789589191 with bad export cookie 9310345924788843490 [10637.321900] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10637.332412] LustreError: 254705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10637.879126] Lustre: server umount lustre-MDT0001 complete [10652.777497] Lustre: server umount lustre-OST0000 complete [10656.739150] Lustre: 250579:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789589194/real 1789589194] req@ffff8a35ef88f800 x1876520086879360/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789589210 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10656.779457] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10656.792275] Lustre: Skipped 2 previous similar messages [10657.169790] Lustre: server umount lustre-OST0001 complete [10673.104543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10683.883958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10693.342986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10704.325518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10724.770416] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [10734.930389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10734.952628] Lustre: lustre-MDT0000: reset Object Index mappings [10735.360637] LustreError: 260590:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10735.373672] LustreError: 260590:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10735.428830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10739.978693] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10740.709319] LustreError: 260590:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10745.829317] LustreError: 260591:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10748.900233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10748.924664] Lustre: lustre-MDT0001: reset Object Index mappings [10749.425826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10753.526574] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10756.862277] Lustre: 261732:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10762.870474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10770.305839] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10774.444739] LustreError: 262086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10779.636115] LustreError: 262087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10780.660175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10782.011589] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10782.281608] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10782.287133] Lustre: Skipped 1 previous similar message [10787.319307] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10789.413401] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10796.995370] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10800.484182] Lustre: 263602:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10806.615871] Lustre: server umount lustre-MDT0000 complete [10807.790438] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10807.796889] LustreError: 260586:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10807.802062] Lustre: Skipped 1 previous similar message [10807.844631] LustreError: 260586:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10810.170609] LustreError: 260571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789589364 with bad export cookie 9310345924788850287 [10810.177346] LustreError: 260571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10810.178884] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10810.552381] Lustre: server umount lustre-MDT0001 complete [10823.312260] Lustre: server umount lustre-OST0000 complete [10837.613407] Lustre: server umount lustre-OST0001 complete [10862.489924] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 16:10:13 (1789589413) [10879.967459] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [10889.780455] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [10926.809316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10927.107335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10927.134795] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10927.239919] Lustre: lustre-MDT0000: new disk, initializing [10927.342287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10927.359760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10932.960675] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10944.307580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10944.423959] Lustre: 267858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10944.505497] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10944.507696] Lustre: Skipped 1 previous similar message [10944.599648] Lustre: lustre-MDT0001: new disk, initializing [10944.757533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10944.769893] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10949.234680] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10959.265793] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10967.078285] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10978.552473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10978.919176] Lustre: lustre-OST0000: new disk, initializing [10978.921919] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10978.940652] Lustre: 269829:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10980.775915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10980.797059] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10980.998547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10988.156415] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11000.772545] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11006.226686] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11006.414800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11010.966308] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11011.199745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11016.676439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11016.679718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11016.683705] Lustre: Skipped 2 previous similar messages [11016.687224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11020.195194] Lustre: server umount lustre-OST0000 complete [11026.916843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11026.927372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11026.931847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11026.941967] Lustre: Skipped 2 previous similar messages [11026.953388] Lustre: Skipped 2 previous similar messages [11032.012079] Lustre: server umount lustre-MDT0000 complete [11032.050284] LustreError: 271171:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11032.069665] LustreError: 271171:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11037.127392] LustreError: 267851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789589591 with bad export cookie 9310345924788852380 [11037.133177] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11037.138614] LustreError: 267851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11037.584447] Lustre: server umount lustre-MDT0001 complete [11061.723792] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [11071.682468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11072.150951] LustreError: 272765:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11072.173566] LustreError: 272765:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11072.232209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11072.235266] Lustre: Skipped 2 previous similar messages [11078.969467] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11090.049761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11095.659959] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11098.889431] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11106.066110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11111.952558] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11121.024641] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11121.202750] Lustre: lustre-OST0001: new disk, initializing [11121.206370] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11121.212442] Lustre: 275075:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11122.464106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11122.479367] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11122.628111] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11128.313159] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11138.397850] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11142.438810] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11148.325590] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11148.395956] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11159.378945] Lustre: server umount lustre-MDT0000 complete [11162.093671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11162.108158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11162.140853] LustreError: 273494:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11162.158564] LustreError: 273494:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11163.997241] LustreError: 273902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789589718 with bad export cookie 9310345924788853262 [11164.021663] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11164.037750] LustreError: 273902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11164.569939] Lustre: server umount lustre-MDT0001 complete [11178.223763] Lustre: server umount lustre-OST0000 complete [11191.267386] Lustre: server umount lustre-OST0001 complete [11209.204861] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [11221.399877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11222.077281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11222.091400] Lustre: Skipped 3 previous similar messages [11228.168802] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11236.780780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11242.827589] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11245.647532] Lustre: 279563:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11252.409449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11258.917604] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11267.218515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11269.275941] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11274.791202] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11278.436620] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11288.516187] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11292.009404] Lustre: 281442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11297.925410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11297.975266] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11311.664987] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 16:17:43 (1789589863) [11329.505229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11329.515288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11329.530073] Lustre: Skipped 3 previous similar messages [11329.544610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11333.214625] Lustre: server umount lustre-MDT0000 complete [11336.161555] LustreError: 278421:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11336.175219] LustreError: 278421:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [11336.634879] LustreError: 278402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789589890 with bad export cookie 9310345924788859737 [11336.646969] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11336.654839] LustreError: 278402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11341.283189] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11341.287428] Lustre: Skipped 4 previous similar messages [11342.970744] Lustre: server umount lustre-MDT0001 complete [11352.615340] Lustre: server umount lustre-OST0000 complete [11357.226965] Lustre: server umount lustre-OST0001 complete [11364.315572] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [11375.722794] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [11427.702612] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [11444.844800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11445.424333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11445.467877] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11445.622334] Lustre: lustre-MDT0000: new disk, initializing [11445.796127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11451.640906] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11464.197946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11464.435887] Lustre: 285971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11464.981441] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11470.705982] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11485.303502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11485.706249] Lustre: 287607:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11485.862144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11485.870957] Lustre: Skipped 5 previous similar messages [11487.116482] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11493.024985] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11503.363780] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11503.453741] Lustre: 288479:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11505.122871] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11505.126785] Lustre: Skipped 1 previous similar message [11505.238639] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11509.561530] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11521.017681] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11525.452782] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11525.457473] Lustre: Skipped 1 previous similar message [11541.883950] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 16:21:33 (1789590093) [11546.083886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11546.088353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11546.107287] Lustre: Skipped 6 previous similar messages [11546.585852] Lustre: server umount lustre-MDT0000 complete [11550.109129] LustreError: 288478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789590104 with bad export cookie 9310345924788867626 [11550.113928] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11550.119263] LustreError: 288478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11550.416452] Lustre: server umount lustre-MDT0001 complete [11564.415709] Lustre: server umount lustre-OST0000 complete [11567.583138] Lustre: 250578:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789590105/real 1789590105] req@ffff8a34c2729f80 x1876520087195392/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789590121 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11569.400538] Lustre: server umount lustre-OST0001 complete [11578.014049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11583.484899] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11592.922847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11593.186566] LustreError: 290724:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11593.206177] LustreError: 290724:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11597.460358] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11602.952710] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11607.417959] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11613.373529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11619.555419] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11627.250079] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11643.426066] loop1: detected capacity change from 0 to 204800000 [11643.646796] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11643.667061] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11643.692663] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11648.735368] loop1: detected capacity change from 0 to 204800000 [11654.488570] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11654.622273] LustreError: 290713:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs14139: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11654.633991] LustreError: 290713:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs14139 [11654.640733] LustreError: 290713:0:(mgs_handler.c:517:mgs_target_reg()) fs14139-MDT0001: Error checking target index: rc = -13 [11654.658218] LustreError: fs14139-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11654.669386] LustreError: 294444:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11654.675596] LustreError: 294444:0:(tgt_mount.c:2128:server_put_super()) no obd fs14139-MDT0001 [11654.679412] LustreError: 294444:0:(tgt_mount.c:116:server_deregister_mount()) fs14139-MDT0001 not registered [11654.722389] Lustre: server umount fs14139-MDT0001 complete [11654.725524] LustreError: 294444:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11665.851903] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11666.013192] Lustre: Setting parameter fs14139-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs14139-MDT0001 [11666.100959] Lustre: srv-fs14139-MDT0001: No data found on store. Initialize space. [11666.104859] Lustre: Skipped 4 previous similar messages [11666.211455] Lustre: fs14139-MDT0001: new disk, initializing [11666.214840] Lustre: Skipped 3 previous similar messages [11666.372502] Lustre: srv-fs14139-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11666.381389] Lustre: 294910:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14139-MDT0001-mdtlov: can't create new llog: rc = -115 [11666.389889] LustreError: 294910:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14139-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11671.465739] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11679.524547] Lustre: server umount fs14139-MDT0001 complete [11694.526180] loop1: detected capacity change from 0 to 204800000 [11701.774187] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11702.498345] Lustre: 296479:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14139-MDT0001-mdtlov: can't create new llog: rc = -115 [11702.528784] LustreError: 296479:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14139-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11705.833693] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11708.882470] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11714.919553] Lustre: server umount fs14139-MDT0001 complete [11726.817938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11726.826206] LustreError: Skipped 1 previous similar message [11726.836872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11726.849812] Lustre: Skipped 3 previous similar messages [11726.866502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11726.873203] Lustre: Skipped 3 previous similar messages [11732.849326] Lustre: server umount lustre-MDT0000 complete [11741.574659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11741.723340] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11747.081824] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11747.319474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11754.930224] loop1: detected capacity change from 0 to 204800000 [11759.972647] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11760.384694] Lustre: 298923:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14139-MDT0001-mdtlov: can't create new llog: rc = -115 [11760.391719] LustreError: 298923:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14139-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11764.486542] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11769.066482] Lustre: server umount fs14139-MDT0001 complete [11792.867104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11792.878342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11792.891698] Lustre: Skipped 4 previous similar messages [11797.357702] Lustre: server umount lustre-OST0000 complete [11806.801186] Lustre: server umount lustre-MDT0000 complete [11810.516509] LustreError: 291468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789590364 with bad export cookie 9310345924788870307 [11810.522844] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11810.528082] LustreError: 291468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11810.925516] Lustre: server umount lustre-MDT0001 complete [11821.299967] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [11823.897161] Key type lgssc unregistered [11824.142905] LNet: 301266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11824.149773] LNetError: 301266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11824.165915] LNet: Removed LNI 192.168.201.148@tcp [11824.899154] Key type .llcrypt unregistered [11824.900982] Key type ._llcrypt unregistered [11842.167267] Key type ._llcrypt registered [11842.168358] Key type .llcrypt registered [11842.255305] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [11843.407488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11843.490690] alg: No test for adler32 (adler32-zlib) [11844.517801] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [11844.744765] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [11846.391596] Key type lgssc registered [11847.477534] Lustre: Echo OBD driver; http://www.lustre.org/ [11856.156646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11856.209607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11857.585884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11862.044286] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11867.108523] LustreError: 302531:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11867.131760] LustreError: 302531:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11870.675978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11871.163779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11875.808453] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11883.676572] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11889.913429] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11896.976565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11897.628915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11902.764111] LustreError: 304358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11906.528666] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11908.074394] LustreError: 304594:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11913.210543] LustreError: 304359:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11915.642921] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11918.928667] LustreError: 304594:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11918.959979] LustreError: 304594:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11929.136198] LustreError: 304358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11929.153615] LustreError: 304358:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11940.553817] loop1: detected capacity change from 0 to 204800000 [11940.957054] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11940.993645] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11941.028179] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11947.004870] LustreError: 304594:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11947.060409] LustreError: 304594:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [11949.502322] loop1: detected capacity change from 0 to 204800000 [11954.648229] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11954.799709] Lustre: fs23811-OST0000: new disk, initializing [11954.801711] Lustre: srv-fs23811-OST0000: No data found on store. Initialize space. [11954.805333] Lustre: 306268:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11954.858484] Lustre: fs23811-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11961.312881] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11971.123883] Lustre: server umount fs23811-OST0000 complete [11980.337138] LustreError: 304359:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11980.363952] LustreError: 304359:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 18 previous similar messages [11982.842361] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11985.343974] loop1: detected capacity change from 0 to 204800000 [11993.231581] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11993.685795] Lustre: fs23811-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11998.181217] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12002.347961] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12013.416455] loop2: detected capacity change from 0 to 204800000 [12013.551475] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12013.557631] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12013.566765] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12019.555949] loop2: detected capacity change from 0 to 204800000 [12026.810415] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12026.913718] LustreError: fs23811-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12026.923072] LustreError: fs23811-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12026.936551] LustreError: 309310:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [12026.943743] LustreError: 309310:0:(tgt_mount.c:2128:server_put_super()) no obd fs23811-OST0001 [12026.950330] LustreError: 309310:0:(tgt_mount.c:116:server_deregister_mount()) fs23811-OST0001 not registered [12027.004196] Lustre: server umount fs23811-OST0001 complete [12027.009133] LustreError: 309310:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12038.836382] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12039.053683] Lustre: fs23811-OST0001: new disk, initializing [12039.073935] Lustre: srv-fs23811-OST0001: No data found on store. Initialize space. [12039.084643] Lustre: 309860:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12039.200553] Lustre: fs23811-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12046.439708] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12046.904251] LustreError: 304360:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12046.936878] LustreError: 304360:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [12056.066710] Lustre: server umount fs23811-OST0000 complete [12068.214890] Lustre: server umount fs23811-OST0001 complete [12083.591562] loop1: detected capacity change from 0 to 204800000 [12089.925286] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12090.216345] Lustre: fs23811-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12097.523138] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12104.336807] loop2: detected capacity change from 0 to 204800000 [12110.744719] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12118.964836] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12129.164474] Lustre: server umount fs23811-OST0000 complete [12136.458076] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12140.830306] Lustre: server umount fs23811-OST0001 complete [12151.782253] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12156.898033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12156.905031] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12156.922270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12156.928208] Lustre: Skipped 1 previous similar message [12156.941558] Lustre: Skipped 1 previous similar message [12161.884317] Lustre: server umount lustre-OST0000 complete [12168.173116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12168.182245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12168.195150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12172.790677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12172.800478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12172.853318] Lustre: Skipped 1 previous similar message [12173.975456] Lustre: server umount lustre-MDT0000 complete [12176.993360] LustreError: 302531:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12177.004650] LustreError: 302531:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 65 previous similar messages [12179.255176] LustreError: 314283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789590733 with bad export cookie 1365514073829257882 [12179.256233] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12179.263638] LustreError: 314283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12179.693463] Lustre: server umount lustre-MDT0001 complete [12194.012451] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12196.607180] Key type lgssc unregistered [12196.917792] LNet: 315788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12196.922773] LNetError: 315788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12196.938261] LNet: Removed LNI 192.168.201.148@tcp [12197.801333] Key type .llcrypt unregistered [12197.803234] Key type ._llcrypt unregistered [12240.328375] Key type ._llcrypt registered [12240.330851] Key type .llcrypt registered [12240.545513] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [12262.990206] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [12264.552859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12264.636123] alg: No test for adler32 (adler32-zlib) [12265.899400] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [12266.360229] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [12268.111930] Key type lgssc registered [12269.933358] Lustre: Echo OBD driver; http://www.lustre.org/ [12310.926547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12310.953395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12312.358400] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12312.425121] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12312.622608] Lustre: lustre-MDT0000: new disk, initializing [12312.846624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12312.872315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12318.868562] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12330.996631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12331.143441] Lustre: 319651:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12331.244135] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12331.250663] Lustre: Skipped 1 previous similar message [12331.351135] Lustre: lustre-MDT0001: new disk, initializing [12331.518102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12331.579797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12331.605299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12337.551865] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12347.931663] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12354.020728] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12361.538582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12361.789742] Lustre: lustre-OST0000: new disk, initializing [12361.793189] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12361.798891] Lustre: 321622:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12361.894047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12369.421434] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12370.026791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12370.044227] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12370.172737] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12381.691629] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12384.265489] Lustre: Mounted lustre-client - version 2.17.58_39_gf14a6ef [12385.500148] Lustre: Unmounted lustre-client [12399.589193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12399.601960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12399.612735] Lustre: Skipped 1 previous similar message [12402.972836] Lustre: server umount lustre-OST0000 complete [12408.288598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12408.296738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12408.311133] Lustre: Skipped 1 previous similar message [12408.320653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12409.827472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12409.828718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12414.944396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12414.954347] Lustre: Skipped 1 previous similar message [12419.040723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12419.060556] Lustre: Skipped 1 previous similar message [12421.087280] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12421.334251] Lustre: server umount lustre-MDT0000 complete [12425.970971] LustreError: 319644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789590980 with bad export cookie 14944860260972917386 [12425.980960] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12425.986752] LustreError: 319644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12426.712457] Lustre: server umount lustre-MDT0001 complete [12435.359930] LNet: 1 local NIs in recovery (showing 1): 192.168.201.148@tcp [12442.286341] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12445.648995] Key type lgssc unregistered [12445.978257] LNet: 323785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12446.007901] LNetError: 323785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12514.669260] LNet: Removed LNI 192.168.201.148@tcp [12515.913591] Key type .llcrypt unregistered [12515.919117] Key type ._llcrypt unregistered [12537.757109] Key type ._llcrypt registered [12537.761272] Key type .llcrypt registered [12537.914361] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [12556.656934] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [12558.091110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12558.133347] alg: No test for adler32 (adler32-zlib) [12559.543159] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [12559.756850] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [12561.487228] Key type lgssc registered [12563.134881] Lustre: Echo OBD driver; http://www.lustre.org/ [12599.624225] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 16:39:10 (1789591150) [12608.103526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12608.134296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12609.395901] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12609.457435] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12609.582469] Lustre: lustre-MDT0000: new disk, initializing [12609.735706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12609.757222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12615.381132] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12627.048132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12627.209738] Lustre: 327008:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12627.281698] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12627.284825] Lustre: Skipped 1 previous similar message [12627.393437] Lustre: lustre-MDT0001: new disk, initializing [12627.601270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12627.668575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12627.678897] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12632.905449] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12642.607509] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12648.609872] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12656.100823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12656.470343] Lustre: lustre-OST0000: new disk, initializing [12656.474464] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12656.480182] Lustre: 328976:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12656.579273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12663.436729] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12666.446488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12666.458378] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12666.644295] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12674.097822] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12692.461434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12692.462297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12692.470941] LustreError: Skipped 1 previous similar message [12692.478770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12692.504900] Lustre: Skipped 1 previous similar message [12692.540702] Lustre: Skipped 1 previous similar message [12696.511908] Lustre: server umount lustre-OST0000 complete [12702.691100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12702.695271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12702.701183] Lustre: Skipped 1 previous similar message [12702.711371] Lustre: Skipped 1 previous similar message [12707.458103] Lustre: server umount lustre-MDT0000 complete [12712.552878] LustreError: 326999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789591266 with bad export cookie 1450697996123815851 [12712.563909] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12713.120790] Lustre: server umount lustre-MDT0001 complete [12724.776629] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12728.153714] Key type lgssc unregistered [12728.491500] LNet: 330883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12728.495880] LNetError: 330883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12728.506810] LNet: Removed LNI 192.168.201.148@tcp [12729.596926] Key type .llcrypt unregistered [12729.599579] Key type ._llcrypt unregistered [12749.906266] Key type ._llcrypt registered [12749.908150] Key type .llcrypt registered [12750.010794] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [12750.987549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12751.085184] alg: No test for adler32 (adler32-zlib) [12752.347595] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [12752.706145] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [12754.479775] Key type lgssc registered [12755.595303] Lustre: Echo OBD driver; http://www.lustre.org/ [12764.313889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12764.341733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12766.373360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12773.442880] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12775.914990] LustreError: 332155:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12775.937814] LustreError: 332155:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12781.026265] LustreError: 332156:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12784.507310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12785.089846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12791.406156] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12799.737411] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12805.443224] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12812.033683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12812.457469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12819.518357] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12827.869216] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12848.645225] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12850.517430] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12856.812238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12856.816779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12856.823205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12856.825388] Lustre: Skipped 1 previous similar message [12859.469808] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.48@tcp (stopping) [12859.478628] Lustre: Skipped 2 previous similar messages [12861.802501] Lustre: server umount lustre-MDT0000 complete [12861.932765] LustreError: 332884:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12861.964663] LustreError: 332884:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12864.570822] LustreError: 332150:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12865.645863] LustreError: 332136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789591419 with bad export cookie 1425705916664112312 [12865.668054] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12867.042747] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12867.056608] Lustre: Skipped 1 previous similar message [12867.061082] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12869.695920] LustreError: 334657:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12869.725790] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.48@tcp (stopping) [12869.728220] LustreError: 334657:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12872.215661] Lustre: server umount lustre-MDT0001 complete [12876.467305] Lustre: server umount lustre-OST0000 complete [12885.840549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12891.018636] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12894.590863] Lustre: 336362:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.148@tcp [12897.567159] Lustre: server umount MGS complete [12907.265134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12907.791744] LustreError: 336934:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12907.817141] LustreError: 336934:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12907.907554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12912.979486] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12922.361613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12922.806866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12927.984076] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12936.951461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12937.451341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12943.948139] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12951.505774] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12953.308278] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12959.534149] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12961.317910] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12966.912552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12966.922615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12966.934531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12966.938456] Lustre: Skipped 1 previous similar message [12968.052335] Lustre: server umount lustre-OST0000 complete [12973.538353] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12973.575324] Lustre: Skipped 1 previous similar message [12978.658891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12978.664725] Lustre: Skipped 3 previous similar messages [12984.737676] Lustre: 331467:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789591522/real 1789591522] req@00000000e1493796 x1876522493343232/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789591538 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12984.766181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12987.360061] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12987.756289] Lustre: server umount lustre-MDT0000 complete [12988.900121] Lustre: 331464:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789591527/real 1789591527] req@00000000a0050015 x1876522493345408/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789591543 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12989.927687] LustreError: 336934:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12989.960855] LustreError: 336934:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [12992.542924] LustreError: 336915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789591546 with bad export cookie 1425705916664113152 [12992.547536] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12992.554501] LustreError: 336915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12993.078558] Lustre: server umount lustre-MDT0001 complete [13005.558338] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [13007.918507] Key type lgssc unregistered [13008.152726] LNet: 340370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13008.157776] LNetError: 340370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13008.180595] LNet: Removed LNI 192.168.201.148@tcp [13009.037750] Key type .llcrypt unregistered [13009.039922] Key type ._llcrypt unregistered [13031.798536] Key type ._llcrypt registered [13031.800746] Key type .llcrypt registered [13031.974237] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [13050.824884] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [13051.528569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13051.667660] alg: No test for adler32 (adler32-zlib) [13052.867269] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [13053.136829] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [13054.784397] Key type lgssc registered [13056.082214] Lustre: Echo OBD driver; http://www.lustre.org/ [13095.165501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13095.192924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13096.529105] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13096.571413] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13096.680546] Lustre: lustre-MDT0000: new disk, initializing [13096.786085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13096.803217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13101.870275] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [13113.892821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13114.042115] Lustre: 343534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13114.133630] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13114.140852] Lustre: Skipped 1 previous similar message [13114.306610] Lustre: lustre-MDT0001: new disk, initializing [13114.436266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13114.494494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13114.506057] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13119.562840] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [13129.889888] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13135.693637] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13143.074163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13143.355261] Lustre: lustre-OST0000: new disk, initializing [13143.359896] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13143.366835] Lustre: 345498:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13143.444078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13149.776458] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [13151.322494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13151.329558] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13151.401074] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13159.322884] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13164.381895] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13164.548215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13169.197760] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13169.415258] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13172.200167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13172.200884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13172.215704] LustreError: Skipped 1 previous similar message [13172.216168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13172.246677] Lustre: Skipped 1 previous similar message [13177.426970] Lustre: server umount lustre-OST0000 complete [13182.441104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13182.445298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13182.456069] Lustre: Skipped 1 previous similar message [13182.464109] Lustre: Skipped 1 previous similar message [13187.249971] Lustre: server umount lustre-MDT0000 complete [13187.560880] LustreError: 343544:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13187.577682] LustreError: 343544:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13191.417704] LustreError: 343524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789591745 with bad export cookie 1798307757316886602 [13191.426839] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13191.431214] LustreError: 343524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13191.756737] Lustre: server umount lustre-MDT0001 complete [13201.684382] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 16:49:12 (1789591752) [13224.138989] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [13237.364741] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [13273.702560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13273.902958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13273.922438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13274.018349] Lustre: lustre-MDT0000: new disk, initializing [13274.087333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13274.100139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13279.434846] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [13290.645982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13290.860741] Lustre: 350260:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13290.941201] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13290.945169] Lustre: Skipped 1 previous similar message [13291.090775] Lustre: lustre-MDT0001: new disk, initializing [13291.261893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13291.320448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13291.326710] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13296.606917] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [13306.248411] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13312.571139] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13319.763443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13320.108338] Lustre: lustre-OST0000: new disk, initializing [13320.113953] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13320.121169] Lustre: 352232:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13320.190332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13326.758714] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [13330.003991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13330.010544] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13330.145903] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13336.576214] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13366.239759] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13366.242933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13366.247548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13366.250038] Lustre: Skipped 2 previous similar messages [13369.312567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13369.319522] Lustre: Skipped 1 previous similar message [13371.821622] Lustre: server umount lustre-OST0000 complete [13378.528648] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13378.536978] LustreError: Skipped 1 previous similar message [13378.539947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13378.553148] Lustre: Skipped 1 previous similar message [13378.563271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13382.183408] Lustre: server umount lustre-MDT0000 complete [13384.677666] LustreError: 351456:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13384.725367] LustreError: 351456:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13388.262654] LustreError: 350251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789591942 with bad export cookie 1798307757316887484 [13388.276299] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13388.288202] LustreError: 350251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13389.116056] Lustre: server umount lustre-MDT0001 complete [13404.309492] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [13407.066403] Key type lgssc unregistered [13407.394792] LNet: 354383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13407.406861] LNetError: 354383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13407.434628] LNet: Removed LNI 192.168.201.148@tcp [13408.840536] Key type .llcrypt unregistered [13408.845961] Key type ._llcrypt unregistered [13504.290650] Key type ._llcrypt registered [13504.293987] Key type .llcrypt registered [13504.652112] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [13504.989295] Key type .llcrypt unregistered [13504.991149] Key type ._llcrypt unregistered [13592.903456] Key type ._llcrypt registered [13592.906821] Key type .llcrypt registered [13593.054931] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [13593.546680] Key type .llcrypt unregistered [13593.549737] Key type ._llcrypt unregistered [13624.368914] Key type ._llcrypt registered [13624.373419] Key type .llcrypt registered [13624.593955] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [13624.978628] Key type .llcrypt unregistered [13624.981195] Key type ._llcrypt unregistered [13656.083104] Key type ._llcrypt registered [13656.084307] Key type .llcrypt registered [13656.254956] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [13656.584390] Key type .llcrypt unregistered [13656.587243] Key type ._llcrypt unregistered [13739.094666] Key type ._llcrypt registered [13739.097418] Key type .llcrypt registered [13739.283962] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [13739.693568] Key type .llcrypt unregistered [13739.695862] Key type ._llcrypt unregistered [13780.225282] Key type ._llcrypt registered [13780.230415] Key type .llcrypt registered [13780.354335] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [13780.648969] Key type .llcrypt unregistered [13780.651799] Key type ._llcrypt unregistered [13813.416815] Key type ._llcrypt registered [13813.418509] Key type .llcrypt registered [13813.573497] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [13813.945413] Key type .llcrypt unregistered [13813.949971] Key type ._llcrypt unregistered [13847.836536] Key type ._llcrypt registered [13847.840067] Key type .llcrypt registered [13848.003586] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [13848.385399] Key type .llcrypt unregistered [13848.386966] Key type ._llcrypt unregistered [13878.629645] Key type ._llcrypt registered [13878.633246] Key type .llcrypt registered [13878.756731] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [13879.059344] Key type .llcrypt unregistered [13879.060925] Key type ._llcrypt unregistered [13913.520775] Key type ._llcrypt registered [13913.525974] Key type .llcrypt registered [13913.623953] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [13913.905793] Key type .llcrypt unregistered [13913.907889] Key type ._llcrypt unregistered [13991.686974] Key type ._llcrypt registered [13991.689256] Key type .llcrypt registered [13991.858868] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [13992.163304] Key type .llcrypt unregistered [13992.165983] Key type ._llcrypt unregistered [14019.438746] Key type ._llcrypt registered [14019.443026] Key type .llcrypt registered [14019.559585] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [14019.846971] Key type .llcrypt unregistered [14019.849363] Key type ._llcrypt unregistered [14048.769272] Key type ._llcrypt registered [14048.771369] Key type .llcrypt registered [14048.925121] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [14049.232644] Key type .llcrypt unregistered [14049.236715] Key type ._llcrypt unregistered [14106.473714] Key type ._llcrypt registered [14106.481561] Key type .llcrypt registered [14106.612526] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [14123.143472] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [14124.028808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14124.057683] alg: No test for adler32 (adler32-zlib) [14125.225495] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [14125.493580] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [14127.287386] Key type lgssc registered [14128.486168] Lustre: Echo OBD driver; http://www.lustre.org/ [14167.239985] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14167.287302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14168.513090] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14168.542752] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14168.597563] Lustre: tst250fs-MDT0000: new disk, initializing [14168.718326] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14168.748337] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14172.523729] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [14185.199865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14185.341979] Lustre: 374661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14185.480762] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14185.491206] Lustre: Skipped 1 previous similar message [14185.570570] Lustre: tst250fs-MDT0001: new disk, initializing [14185.689303] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14185.803426] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14185.832943] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14193.379408] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [14198.445031] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14206.071621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14206.326717] Lustre: tst250fs-OST0000: new disk, initializing [14206.330866] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14206.337629] Lustre: 376294:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14206.422900] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14210.208147] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14210.212716] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14210.282123] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14213.294604] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [14226.134912] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14226.488487] Lustre: tst250fs-OST0001: new disk, initializing [14226.494945] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14226.501919] Lustre: 377166:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14226.655216] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14232.653104] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14232.661117] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14232.775719] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14234.904178] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [14241.902439] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14244.411580] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14244.419589] Lustre: Skipped 1 previous similar message [14267.882360] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14267.901414] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14267.926705] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14268.898447] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14268.907975] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14268.921466] Lustre: Skipped 2 previous similar messages [14268.934578] Lustre: Skipped 3 previous similar messages [14271.825911] Lustre: server umount tst250fs-MDT0000 complete [14276.175904] LustreError: 375457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789592830 with bad export cookie 14474579330250593560 [14276.180461] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14276.186541] LustreError: 375457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14276.552432] Lustre: server umount tst250fs-MDT0001 complete [14290.172936] Lustre: server umount tst250fs-OST0000 complete [14304.871952] Lustre: server umount tst250fs-OST0001 complete [14327.128434] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [14337.078838] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [14378.281653] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 17:08:48 (1789592928) [14381.668521] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14407.448767] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [14425.775912] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [14460.908075] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14137 sec ========= 17:10:12 (1789593012) [14462.674913] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:10:14 (1789593014) === [14465.720801] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:10:17 (1789593017) === [14509.285278] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [14512.249700] Key type lgssc unregistered [14512.618280] LNet: 386321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14512.626479] LNetError: 386321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14512.651910] LNet: Removed LNI 192.168.201.148@tcp [14513.539219] Key type .llcrypt unregistered [14513.542867] Key type ._llcrypt unregistered