[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 467885119 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002360] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.006835] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.010005] LSM: Security Framework initializing [ 0.011081] Yama: becoming mindful. [ 0.012052] SELinux: Initializing. [ 0.013087] *** VALIDATE selinux *** [ 0.021842] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026799] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028052] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030018] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031160] *** VALIDATE tmpfs *** [ 0.033244] *** VALIDATE proc *** [ 0.034336] *** VALIDATE cgroup *** [ 0.035015] *** VALIDATE cgroup2 *** [ 0.037003] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038179] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041033] Spectre V2 : User space: Vulnerable [ 0.042018] Speculative Store Bypass: Vulnerable [ 0.045463] debug: unmapping init [mem 0xffffffff90259000-0xffffffff90260fff] [ 0.047222] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048808] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049047] ... version: 2 [ 0.050027] ... bit width: 48 [ 0.051023] ... generic registers: 4 [ 0.052022] ... value mask: 0000ffffffffffff [ 0.053023] ... max period: 00007fffffffffff [ 0.054025] ... fixed-purpose events: 3 [ 0.055019] ... event mask: 000000070000000f [ 0.057317] rcu: Hierarchical SRCU implementation. [ 0.059821] smp: Bringing up secondary CPUs ... [ 0.060752] x86: Booting SMP configuration: [ 0.061046] .... node #0, CPUs: #1 #2 #3 [ 0.065107] smp: Brought up 1 node, 4 CPUs [ 0.067023] smpboot: Max logical packages: 1 [ 0.068022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.233269] node 0 deferred pages initialised in 162ms [ 0.236179] devtmpfs: initialized [ 0.237335] x86/mm: Memory block size: 128MB [ 0.239708] gcov: version magic: 0x41383552 [ 0.241396] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.242090] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.243327] pinctrl core: initialized pinctrl subsystem [ 0.244286] [ 0.244886] ************************************************************* [ 0.245021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.246019] ** ** [ 0.247021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.248020] ** ** [ 0.249107] ** This means that this kernel is built to expose internal ** [ 0.250022] ** IOMMU data structures, which may compromise security on ** [ 0.251019] ** your system. ** [ 0.252023] ** ** [ 0.253019] ** If you see this message and you are not debugging the ** [ 0.254020] ** kernel, report this immediately to your vendor! ** [ 0.255020] ** ** [ 0.256023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.257021] ************************************************************* [ 0.258810] NET: Registered protocol family 16 [ 0.259569] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.260086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.261092] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.263023] cpuidle: using governor menu [ 0.264784] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.266354] PCI: Using configuration type 1 for base access [ 0.267096] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.273211] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.274049] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.276046] cryptd: max_cpu_qlen set to 1000 [ 0.278331] ACPI: Added _OSI(Module Device) [ 0.279025] ACPI: Added _OSI(Processor Device) [ 0.280022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.281022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.285277] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.289378] ACPI: Interpreter enabled [ 0.290089] ACPI: PM: (supports S0 S3 S4 S5) [ 0.291026] ACPI: Using IOAPIC for interrupt routing [ 0.292191] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.293436] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.302969] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.303093] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.304030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.305116] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.307469] acpiphp: Slot [2] registered [ 0.308224] acpiphp: Slot [5] registered [ 0.309154] acpiphp: Slot [6] registered [ 0.310152] acpiphp: Slot [7] registered [ 0.311172] acpiphp: Slot [8] registered [ 0.312181] acpiphp: Slot [9] registered [ 0.313225] acpiphp: Slot [10] registered [ 0.314227] acpiphp: Slot [3] registered [ 0.315155] acpiphp: Slot [4] registered [ 0.316173] acpiphp: Slot [11] registered [ 0.317110] acpiphp: Slot [12] registered [ 0.318129] acpiphp: Slot [13] registered [ 0.319113] acpiphp: Slot [14] registered [ 0.320138] acpiphp: Slot [15] registered [ 0.321184] acpiphp: Slot [16] registered [ 0.322139] acpiphp: Slot [17] registered [ 0.323136] acpiphp: Slot [18] registered [ 0.324130] acpiphp: Slot [19] registered [ 0.325129] acpiphp: Slot [20] registered [ 0.326110] acpiphp: Slot [21] registered [ 0.327159] acpiphp: Slot [22] registered [ 0.328152] acpiphp: Slot [23] registered [ 0.329131] acpiphp: Slot [24] registered [ 0.330154] acpiphp: Slot [25] registered [ 0.331203] acpiphp: Slot [26] registered [ 0.332100] acpiphp: Slot [27] registered [ 0.333111] acpiphp: Slot [28] registered [ 0.334131] acpiphp: Slot [29] registered [ 0.336146] acpiphp: Slot [30] registered [ 0.337115] acpiphp: Slot [31] registered [ 0.338070] PCI host bridge to bus 0000:00 [ 0.339013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.341025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.342138] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.344021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.346020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.348055] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.350215] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.351885] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.355074] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.362014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.366047] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.368017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.370020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.372017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.374017] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.377189] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.380048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.382704] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.387018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.397016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.402013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.407890] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.424019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.438025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.466023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.480234] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.489020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.497020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.522018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.533533] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.546020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.554018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.576033] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.589928] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.599018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.607019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.626028] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.638252] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.647017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.654019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.675018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.689474] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.700029] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.708019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.729022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.740554] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.743419] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.746395] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.749441] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.751262] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.756154] iommu: Default domain type: Passthrough [ 0.758544] SCSI subsystem initialized [ 0.760241] ACPI: bus type USB registered [ 0.762125] usbcore: registered new interface driver usbfs [ 0.764087] usbcore: registered new interface driver hub [ 0.766157] usbcore: registered new device driver usb [ 0.768180] pps_core: LinuxPPS API ver. 1 registered [ 0.770015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.773064] PTP clock support registered [ 0.775133] EDAC MC: Ver: 3.0.0 [ 0.777152] PCI: Using ACPI for IRQ routing [ 0.779116] NetLabel: Initializing [ 0.780013] NetLabel: domain hash size = 128 [ 0.781017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.782136] NetLabel: unlabeled traffic allowed by default [ 0.784081] vgaarb: loaded [ 0.785275] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.787022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.793275] clocksource: Switched to clocksource kvm-clock [ 0.897734] VFS: Disk quotas dquot_6.6.0 [ 0.899328] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.901863] *** VALIDATE ramfs *** [ 0.903179] *** VALIDATE hugetlbfs *** [ 0.904785] pnp: PnP ACPI init [ 0.907285] pnp: PnP ACPI: found 6 devices [ 0.926704] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.929931] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.932169] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.934396] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.936754] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.939193] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.941960] NET: Registered protocol family 2 [ 0.944262] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.948917] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.952673] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.959102] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.962562] TCP: Hash tables configured (established 65536 bind 65536) [ 0.965291] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.968223] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.971348] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.974613] NET: Registered protocol family 1 [ 0.977450] RPC: Registered named UNIX socket transport module. [ 0.979742] RPC: Registered udp transport module. [ 0.981593] RPC: Registered tcp transport module. [ 0.983074] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.986892] NET: Registered protocol family 44 [ 0.988896] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.991371] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.994429] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.997073] PCI: CLS 0 bytes, default 64 [ 1.000267] Unpacking initramfs... [ 2.432620] debug: unmapping init [mem 0xffff89c5bcc54000-0xffff89c5bffbffff] [ 2.436308] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.438062] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.440404] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.907908] Initialise system trusted keyrings [ 2.909744] Key type blacklist registered [ 2.911779] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.920415] zbud: loaded [ 2.923169] *** VALIDATE nfs *** [ 2.924496] *** VALIDATE nfs4 *** [ 2.926053] pstore: using deflate compression [ 2.929707] Platform Keyring initialized [ 3.018459] NET: Registered protocol family 38 [ 3.020429] Key type asymmetric registered [ 3.022017] Asymmetric key parser 'x509' registered [ 3.023890] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.027288] io scheduler mq-deadline registered [ 3.028931] io scheduler kyber registered [ 3.030515] io scheduler bfq registered [ 3.031735] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.033801] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.035477] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.037308] ACPI: Power Button [PWRF] [ 3.042300] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.049349] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.063961] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.071482] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.087694] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.115436] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.144403] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.149322] Non-volatile memory driver v1.3 [ 3.150842] Linux agpgart interface v0.103 [ 3.184081] virtio_blk virtio1: [vda] 146856 512-byte logical blocks (75.2 MB/71.7 MiB) [ 3.187285] vda: detected capacity change from 0 to 75190272 [ 3.205064] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.207144] vdb: detected capacity change from 0 to 1073741824 [ 3.221417] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.223440] vdc: detected capacity change from 0 to 2621440000 [ 3.243721] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.246987] vdd: detected capacity change from 0 to 2621440000 [ 3.267349] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.270682] vde: detected capacity change from 0 to 4294967296 [ 3.297074] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.299870] vdf: detected capacity change from 0 to 4294967296 [ 3.307504] libphy: Fixed MDIO Bus: probed [ 3.320453] usbcore: registered new interface driver usbserial_generic [ 3.323120] usbserial: USB Serial support registered for generic [ 3.325822] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.330650] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.332717] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.335499] mousedev: PS/2 mouse device common for all mice [ 3.337600] rtc_cmos 00:05: RTC can wake from S4 [ 3.339859] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.340070] rtc_cmos 00:05: registered as rtc0 [ 3.343805] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.346878] intel_pstate: CPU model not supported [ 3.349679] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.354197] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.355682] hid: raw HID events driver (C) Jiri Kosina [ 3.359952] usbcore: registered new interface driver usbhid [ 3.362123] usbhid: USB HID core driver [ 3.363823] drop_monitor: Initializing network drop monitor service [ 3.366397] Initializing XFRM netlink socket [ 3.368500] NET: Registered protocol family 10 [ 3.371482] Segment Routing with IPv6 [ 3.372954] NET: Registered protocol family 17 [ 3.375383] mpls_gso: MPLS GSO support [ 3.380162] RAS: Correctable Errors collector initialized. [ 3.382163] AVX version of gcm_enc/dec engaged. [ 3.383628] AES CTR mode by8 optimization enabled [ 3.451437] sched_clock: Marking stable (3451383030, 0)->(4423342539, -971959509) [ 3.454568] registered taskstats version 1 [ 3.456292] Loading compiled-in X.509 certificates [ 3.458045] zswap: loaded using pool lzo/zbud [ 3.482431] Key type big_key registered [ 3.494904] Key type encrypted registered [ 3.496456] ima: No TPM chip found, activating TPM-bypass! [ 3.498282] ima: Allocated hash algorithm: sha1 [ 3.499795] ima: No architecture policies found [ 3.501255] evm: Initialising EVM extended attributes: [ 3.502920] evm: security.selinux [ 3.504014] evm: security.ima [ 3.504932] evm: security.capability [ 3.506049] evm: HMAC attrs: 0x1 [ 3.508301] rtc_cmos 00:05: setting system clock to 2026-08-27 18:43:00 UTC (1787856180) [ 3.514490] debug: unmapping init [mem 0xffffffff91203000-0xffffffff913fffff] [ 3.517109] debug: unmapping init [mem 0xffffffff8ff82000-0xffffffff90258fff] [ 3.525107] Write protecting the kernel read-only data: 28672k [ 3.528248] debug: unmapping init [mem 0xffffffff8e603000-0xffffffff8e7fffff] [ 3.530872] debug: unmapping init [mem 0xffffffff8ef14000-0xffffffff8effffff] [ 3.564975] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.572265] systemd[1]: Detected virtualization kvm. [ 3.573927] systemd[1]: Detected architecture x86-64. [ 3.575487] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.623640] systemd[1]: No hostname configured. [ 3.625569] systemd[1]: Set hostname to . [ 3.629387] random: systemd: uninitialized urandom read (16 bytes read) [ 3.632203] systemd[1]: Initializing machine ID from random generator. [ 3.703458] random: ln: uninitialized urandom read (6 bytes read) [ 3.776567] random: systemd: uninitialized urandom read (16 bytes read) [ 3.779895] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.786609] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ 3.790350] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.476794] device-mapper: uevent: version 1.0.3 [ 4.479217] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.250745] virtio_net virtio0 ens2: renamed from eth0 [ 5.257714] random: fast init done [ 5.290289] scsi host0: ata_piix [ 5.318358] scsi host1: ata_piix [ 5.320273] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.322859] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.795935] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.010701] random: crng init done [ 10.012317] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.539450] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.752555] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.002735] SELinux: Disabled at runtime. [ 12.063473] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.073705] systemd[1]: Detected virtualization kvm. [ 12.075803] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.579761] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.583515] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.588120] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.593456] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.597541] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.605112] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.613277] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ 12.790468] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.144309] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.486305] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.565381] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.718956] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.733874] EDAC sbridge: Ver: 1.1.2 [ 15.575276] Key type dns_resolver registered [ 15.880448] NFS: Registering the id_resolver key type [ 15.882853] Key type id_resolver registered [ 15.885020] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg235-server login: [ 41.731201] libcfs: loading out-of-tree module taints kernel. [ 41.747849] Key type ._llcrypt registered [ 41.749067] Key type .llcrypt registered [ 41.794976] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 62.907328] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 64.384025] hrtimer: interrupt took 6030019 ns [ 64.814545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 64.853969] alg: No test for adler32 (adler32-zlib) [ 66.219679] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 67.348116] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 69.056186] Key type lgssc registered [ 70.695656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 90.984994] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 136.058393] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 150.136092] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 150.179476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 151.619926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 151.664337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 151.862576] Lustre: lustre-MDT0000: new disk, initializing [ 152.032695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 152.075297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 156.548620] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.707822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 170.820297] Lustre: 6499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 170.852787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 170.858963] Lustre: Skipped 1 previous similar message [ 170.933813] Lustre: lustre-MDT0001: new disk, initializing [ 170.974994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 171.038072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 171.068845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 176.138608] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.798183] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 192.489285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 192.715217] Lustre: lustre-OST0000: new disk, initializing [ 192.718173] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 192.724871] Lustre: 8437:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 192.782877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.430343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 196.438360] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 196.498084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 198.785715] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.211241] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 213.331303] Lustre: lustre-OST0001: new disk, initializing [ 213.334848] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 213.340198] Lustre: 9510:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 213.400119] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 219.202135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 219.215502] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 219.323756] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 219.371706] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.122987] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 240.112251] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 248.101060] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing check_logdir /tmp/testlogs/ [ 253.497883] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing yml_node [ 258.620528] Lustre: DEBUG MARKER: Client: 2.17.57.81 [ 261.367799] Lustre: DEBUG MARKER: MDS: 2.17.57.81 [ 264.346824] Lustre: DEBUG MARKER: OSS: 2.17.57.81 [ 266.178947] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 14:47:21 EDT 2026 [ 284.828706] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 286.743496] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 304.097409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 304.102418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.141053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.659301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.660357] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.679036] Lustre: Skipped 2 previous similar messages [ 306.718048] Lustre: Skipped 2 previous similar messages [ 310.573693] Lustre: server umount lustre-MDT0000 complete [ 316.172128] LustreError: 6493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856493 with bad export cookie 15060774274363862084 [ 316.176935] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 316.201872] LustreError: 6493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 316.897028] LustreError: 8445:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.899233] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.931492] LustreError: 8445:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 316.975730] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 316.990252] Lustre: Skipped 2 previous similar messages [ 322.018636] LustreError: 6506:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 322.019181] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 322.067321] Lustre: Skipped 1 previous similar message [ 322.714932] Lustre: server umount lustre-MDT0001 complete [ 327.845645] Lustre: server umount lustre-OST0000 complete [ 331.605118] Lustre: server umount lustre-OST0001 complete [ 341.064459] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 352.881795] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 391.525850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 391.795838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 391.861783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 391.987462] Lustre: lustre-MDT0000: new disk, initializing [ 392.121137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 392.157622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 397.723741] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 409.518778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 409.698150] Lustre: 15068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 409.771189] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 409.775296] Lustre: Skipped 1 previous similar message [ 409.886031] Lustre: lustre-MDT0001: new disk, initializing [ 410.031508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 410.076510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 410.095250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 415.049728] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 425.274995] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 431.127294] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 438.442203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 438.741475] Lustre: lustre-OST0000: new disk, initializing [ 438.746684] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 438.755281] Lustre: 17034:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 438.849662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 439.981293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 439.995279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 440.218034] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 445.585164] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 455.072844] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 459.681981] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 459.934943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 463.969470] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 464.164368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 471.015815] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 471.024876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 471.032275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 471.047335] Lustre: Skipped 2 previous similar messages [ 471.063811] Lustre: Skipped 1 previous similar message [ 480.225754] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 480.420307] Lustre: server umount lustre-OST0000 complete [ 486.381687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 486.386881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 486.395849] Lustre: Skipped 1 previous similar message [ 486.408647] Lustre: Skipped 3 previous similar messages [ 490.798715] Lustre: server umount lustre-MDT0000 complete [ 491.496601] LustreError: 15073:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 491.513596] LustreError: 15073:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 494.838972] LustreError: 15060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856671 with bad export cookie 15060774274363863617 [ 494.844161] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 494.857479] LustreError: 15060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 495.433742] Lustre: server umount lustre-MDT0001 complete [ 502.830072] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 14:51:17 (1787856677) [ 525.919464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 526.553423] LustreError: 19913:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.670974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 531.857759] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 531.940039] LustreError: 19914:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.162537] LustreError: 19913:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.175770] LustreError: 19913:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 542.254499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.737165] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 556.294232] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 562.518631] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 564.998499] Lustre: Failing over lustre-MDT0000 [ 567.074182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 567.091466] LustreError: 19913:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.112354] LustreError: 19913:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 567.240855] Lustre: server umount lustre-MDT0000 complete [ 583.584371] Lustre: 3634:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787856744/real 1787856744] req@ffff89c635d96940 x1874703149073920/t0(0) o400->MGC192.168.202.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787856760 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 583.618178] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 587.693201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 594.245868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 594.257902] Lustre: Skipped 1 previous similar message [ 599.536070] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 599.555860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 599.581074] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 599.828343] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 610.272934] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 614.881570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 614.896958] LustreError: Skipped 1 previous similar message [ 614.903554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 614.921942] Lustre: Skipped 1 previous similar message [ 614.932248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 620.735545] Lustre: server umount lustre-MDT0000 complete [ 623.076057] LustreError: 19908:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 623.106656] LustreError: 19908:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 15 previous similar messages [ 624.783948] LustreError: 19895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856801 with bad export cookie 15060774274363865031 [ 624.800189] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 625.123036] Lustre: server umount lustre-MDT0001 complete [ 633.900739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 639.216577] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 648.298767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 654.245953] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 661.466795] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 666.978877] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 669.502700] Lustre: Failing over lustre-MDT0000 [ 669.520244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 669.542438] Lustre: Skipped 1 previous similar message [ 669.558717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 669.564569] Lustre: Skipped 2 previous similar messages [ 669.870183] Lustre: server umount lustre-MDT0000 complete [ 689.122248] Lustre: 3632:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787856850/real 1787856850] req@ffff89c61a3d7840 x1874703149103872/t0(0) o400->MGC192.168.202.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787856866 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 689.132590] LustreError: 23253:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 689.175296] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 689.223557] LustreError: 23253:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 10 previous similar messages [ 691.413787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 700.427071] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd1029e6839ccbb71 [ 700.454798] LustreError: 25256:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.202.135@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (ffff89c604bf8600) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 700.502626] Lustre: MGC192.168.202.135@tcp: Connection restored to 0@lo (at 0@lo) [ 700.524918] Lustre: Skipped 1 previous similar message [ 701.277441] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 706.536985] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 706.549904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 706.600258] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 708.349240] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 718.403944] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 728.865203] Lustre: server umount lustre-MDT0000 complete [ 733.239136] LustreError: 23234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856910 with bad export cookie 15060774274363865969 [ 733.242378] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 733.252279] LustreError: 23234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 733.666749] Lustre: server umount lustre-MDT0001 complete [ 742.916942] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 14:55:17 (1787856917) [ 761.478694] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 773.928412] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 820.310205] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 830.988559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 831.255774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 831.285272] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 831.375244] Lustre: lustre-MDT0000: new disk, initializing [ 831.475278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 831.480305] Lustre: Skipped 3 previous similar messages [ 831.515819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 837.131947] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 848.900901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 849.085835] Lustre: 29628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 849.520214] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 856.178797] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 868.580868] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 876.904826] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 886.405838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 886.741950] Lustre: 31600:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 887.890666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 887.957570] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 888.065907] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 888.088458] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 893.399629] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 903.833193] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 909.951302] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 934.373858] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 934.377890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 934.379139] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 934.397692] Lustre: Skipped 4 previous similar messages [ 934.417661] Lustre: Skipped 3 previous similar messages [ 937.733870] Lustre: server umount lustre-OST0000 complete [ 956.384725] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 956.620578] Lustre: server umount lustre-MDT0000 complete [ 959.977182] LustreError: 30814:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 960.000833] LustreError: 30814:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 11 previous similar messages [ 960.058649] LustreError: 29621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857137 with bad export cookie 15060774274363866382 [ 960.083316] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 960.409962] Lustre: server umount lustre-MDT0001 complete [ 975.515907] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 979.372861] Key type lgssc unregistered [ 979.821161] LNet: 33584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.838892] LNetError: 33584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.864962] LNet: Removed LNI 192.168.202.135@tcp [ 980.970190] Key type .llcrypt unregistered [ 980.972195] Key type ._llcrypt unregistered [ 1010.957200] Key type ._llcrypt registered [ 1010.958796] Key type .llcrypt registered [ 1011.046728] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 1012.423910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1012.465435] alg: No test for adler32 (adler32-zlib) [ 1013.498849] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 1013.689936] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 1015.320260] Key type lgssc registered [ 1016.465303] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1023.900692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1023.950204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1025.525435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1030.200731] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1035.754883] LustreError: 35000:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1035.780069] LustreError: 35000:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1039.298701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1039.924111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1044.595995] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1053.453591] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1059.285959] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1066.882861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1067.323712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1070.416711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1074.316530] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1082.093235] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1084.748237] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1099.476650] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 15:01:14 (1787857274) [ 1101.597656] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1103.435742] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1110.692394] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 15:01:25 (1787857285) [ 1119.722407] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 15:01:34 (1787857294) [ 1121.045134] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1132.063307] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1132.070323] Lustre: Skipped 2 previous similar messages [ 1133.893425] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 15:01:48 (1787857308) [ 1136.514664] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1136.534382] Lustre: Skipped 2 previous similar messages [ 1145.907221] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1145.914156] Lustre: Skipped 4 previous similar messages [ 1163.134535] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 15:02:17 (1787857337) [ 1166.553704] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1170.357889] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1180.199471] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 15:02:35 (1787857355) [ 1182.431832] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1182.436324] Lustre: Skipped 6 previous similar messages [ 1197.005890] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 15:02:51 (1787857371) [ 1212.899625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1212.930690] Lustre: Skipped 1 previous similar message [ 1212.933381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1212.937343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1218.018286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1218.031600] Lustre: Skipped 2 previous similar messages [ 1218.286846] Lustre: server umount lustre-OST0000 complete [ 1223.138751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1223.142523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1223.149570] Lustre: Skipped 1 previous similar message [ 1228.257563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1228.267514] Lustre: Skipped 2 previous similar messages [ 1228.989529] Lustre: server umount lustre-MDT0000 complete [ 1233.249446] LustreError: 36841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857410 with bad export cookie 9796857902857429372 [ 1233.270562] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1233.289755] LustreError: 36841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1234.095362] Lustre: server umount lustre-MDT0001 complete [ 1247.817601] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 1250.974908] Key type lgssc unregistered [ 1251.337924] LNet: 42873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1251.344606] LNetError: 42873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1251.356898] LNet: Removed LNI 192.168.202.135@tcp [ 1252.485181] Key type .llcrypt unregistered [ 1252.487730] Key type ._llcrypt unregistered [ 1273.612932] Key type ._llcrypt registered [ 1273.615233] Key type .llcrypt registered [ 1273.699340] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 1290.524879] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 1291.949733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1292.045375] alg: No test for adler32 (adler32-zlib) [ 1293.343615] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 1293.696680] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 1295.472386] Key type lgssc registered [ 1296.999448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1332.337227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1332.378946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1333.633526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1333.656323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1333.709891] Lustre: lustre-MDT0000: new disk, initializing [ 1333.782506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1333.802142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1338.391322] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1348.207432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.319122] Lustre: 46030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1348.371571] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1348.375669] Lustre: Skipped 1 previous similar message [ 1348.443477] Lustre: lustre-MDT0001: new disk, initializing [ 1348.538439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1348.586669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1348.596676] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1353.467651] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1362.285255] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1368.627081] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1375.685416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1376.123527] Lustre: lustre-OST0000: new disk, initializing [ 1376.128812] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1376.133379] Lustre: 47995:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1376.239223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1383.023498] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1386.574929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1386.591230] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1386.670614] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1392.175718] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1397.247185] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1397.490634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1403.499680] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1403.805940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1407.461583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1407.465636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1407.499242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1407.515059] Lustre: Skipped 1 previous similar message [ 1407.558300] Lustre: Skipped 1 previous similar message [ 1412.577912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1412.599097] Lustre: Skipped 1 previous similar message [ 1417.699362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1417.705394] Lustre: Skipped 1 previous similar message [ 1421.280185] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1421.479250] Lustre: server umount lustre-OST0000 complete [ 1427.938118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1427.939932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1427.950358] Lustre: Skipped 1 previous similar message [ 1432.572186] Lustre: server umount lustre-MDT0000 complete [ 1436.817366] LustreError: 46022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857613 with bad export cookie 10040726935513635050 [ 1436.848821] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1437.272376] Lustre: server umount lustre-MDT0001 complete [ 1448.387421] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 15:07:03 (1787857623) [ 1456.336866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1456.940996] LustreError: 50457:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1456.965174] LustreError: 50457:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 1457.031736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1462.038822] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1462.243568] LustreError: 50458:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1462.274758] LustreError: 50458:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1466.362959] LustreError: 50457:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1471.467934] LustreError: 50458:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1471.929214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1472.355057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1477.210445] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1487.431181] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1494.869670] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1501.839294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1502.189754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1508.110479] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1516.260552] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1532.742867] LustreError: 53009:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1534.187646] Lustre: Modifying parameter general.timeout=1 in log params [ 1535.956351] Lustre: Modifying parameter general.timeout=2 in log params [ 1539.295444] Lustre: Modifying parameter general.timeout=4 in log params [ 1539.302977] Lustre: Skipped 1 previous similar message [ 1543.648384] Lustre: Modifying parameter general.timeout=7 in log params [ 1543.650815] Lustre: Skipped 2 previous similar messages [ 1552.110310] Lustre: Modifying parameter general.timeout=14 in log params [ 1552.116732] Lustre: Skipped 6 previous similar messages [ 1569.017274] Lustre: Modifying parameter general.timeout=31 in log params [ 1569.025302] Lustre: Skipped 16 previous similar messages [ 1601.218354] Lustre: Modifying parameter general.timeout=64 in log params [ 1601.220566] Lustre: Skipped 32 previous similar messages [ 1670.681579] Lustre: Disabling parameter general.timeout= in log params [ 1670.684547] Lustre: Skipped 65 previous similar messages [ 1674.407892] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 15:10:49 (1787857849) [ 1677.857396] Lustre: 59963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1679.809067] Lustre: 60062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1683.426784] Lustre: 60278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1683.432987] Lustre: 60278:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1717.474246] Lustre: 61844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1717.483229] Lustre: 61844:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1717.499345] LustreError: 61844:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1722.455809] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 15:11:37 (1787857897) [ 1738.733388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1738.746873] LustreError: Skipped 1 previous similar message [ 1738.756924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1738.778997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1738.784775] Lustre: Skipped 1 previous similar message [ 1742.560727] Lustre: server umount lustre-MDT0000 complete [ 1745.379819] LustreError: 51185:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1745.406888] LustreError: 51185:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1747.214585] LustreError: 50438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857924 with bad export cookie 10040726935513635932 [ 1747.227887] LustreError: 50438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1747.228949] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1747.714759] Lustre: server umount lustre-MDT0001 complete [ 1762.587820] Lustre: server umount lustre-OST0000 complete [ 1802.114473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1802.511577] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1802.570975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1802.579521] Lustre: Skipped 22 previous similar messages [ 1802.947681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1807.602920] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1817.527726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1817.760935] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1817.813910] Lustre: 64376:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1824.751375] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1836.296292] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1845.632757] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1855.037820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1855.302888] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1855.718350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1855.723124] Lustre: Skipped 1 previous similar message [ 1862.137889] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1870.818298] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1873.212278] Lustre: 66945:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1884.136653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1884.165179] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1884.183138] Lustre: Skipped 2 previous similar messages [ 1884.194740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1884.200851] Lustre: Skipped 3 previous similar messages [ 1887.514625] Lustre: server umount lustre-OST0000 complete [ 1894.881802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1894.884773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1894.926017] Lustre: Skipped 2 previous similar messages [ 1898.340629] Lustre: server umount lustre-MDT0000 complete [ 1900.003975] LustreError: 64388:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1900.035953] LustreError: 64388:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1902.784094] LustreError: 65119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858079 with bad export cookie 10040726935513639103 [ 1902.786186] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1902.820806] LustreError: 65119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1903.430949] Lustre: server umount lustre-MDT0001 complete [ 1917.887430] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 1920.917743] Key type lgssc unregistered [ 1921.181774] LNet: 68156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1921.190736] LNetError: 68156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1921.212692] LNet: Removed LNI 192.168.202.135@tcp [ 1922.298207] Key type .llcrypt unregistered [ 1922.301230] Key type ._llcrypt unregistered [ 1951.366967] Key type ._llcrypt registered [ 1951.376377] Key type .llcrypt registered [ 1951.505248] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 1952.421686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1952.499758] alg: No test for adler32 (adler32-zlib) [ 1953.631849] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 1953.945686] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 1955.688316] Key type lgssc registered [ 1956.891991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1964.702117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1964.722154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1966.347632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1971.392435] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1976.813452] LustreError: 69572:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1976.835742] LustreError: 69572:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1981.403110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1982.114478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1986.878928] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 1996.048352] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2001.522686] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2009.365732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2009.966625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2016.569131] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2024.523315] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2035.788207] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2047.976442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2047.982791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2047.993130] Lustre: Skipped 1 previous similar message [ 2048.035058] Lustre: Skipped 2 previous similar messages [ 2051.394590] Lustre: server umount lustre-MDT0000 complete [ 2053.089676] LustreError: 69571:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2053.120485] LustreError: 69571:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2055.547115] LustreError: 69552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858232 with bad export cookie 4287730641749813389 [ 2055.569054] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2056.076948] Lustre: server umount lustre-MDT0001 complete [ 2071.051937] Lustre: server umount lustre-OST0000 complete [ 2099.839717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2100.064946] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2100.092821] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2100.325405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2104.304146] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2115.058654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2115.151829] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2115.230876] Lustre: 74443:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2115.837690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2120.927654] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2129.964111] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2136.019237] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2143.720275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2143.879504] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2144.184042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2152.236397] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2163.803391] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2165.860631] Lustre: 77020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2168.145352] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2179.041134] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2179.050753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2179.063497] Lustre: Skipped 1 previous similar message [ 2179.071728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2179.572178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2180.998130] Lustre: server umount lustre-OST0000 complete [ 2187.233428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2187.243398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2187.276275] Lustre: Skipped 1 previous similar message [ 2187.288203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2187.305788] Lustre: Skipped 1 previous similar message [ 2189.797691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.798857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2189.821994] Lustre: Skipped 1 previous similar message [ 2191.841121] Lustre: server umount lustre-MDT0000 complete [ 2194.916406] LustreError: 74449:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.948261] LustreError: 74449:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2195.994532] LustreError: 76323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858372 with bad export cookie 4287730641749814229 [ 2196.001741] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2196.006846] LustreError: 76323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2196.369924] Lustre: server umount lustre-MDT0001 complete [ 2210.829994] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 2214.002729] Key type lgssc unregistered [ 2214.350570] LNet: 78238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2214.360444] LNetError: 78238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2214.397227] LNet: Removed LNI 192.168.202.135@tcp [ 2215.456125] Key type .llcrypt unregistered [ 2215.459567] Key type ._llcrypt unregistered [ 2247.006695] Key type ._llcrypt registered [ 2247.009136] Key type .llcrypt registered [ 2247.130327] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 2248.952824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2249.054400] alg: No test for adler32 (adler32-zlib) [ 2250.263785] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 2250.672169] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 2252.400199] Key type lgssc registered [ 2253.835217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2262.969095] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2262.978150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2264.655248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2269.770182] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2274.789690] LustreError: 79652:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2274.810339] LustreError: 79652:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2279.911274] LustreError: 79653:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2281.846682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2282.615509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2287.823617] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2297.549842] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2304.552919] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2312.728978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2313.539984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2321.286828] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2331.105881] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2343.745843] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2344.806051] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2345.968120] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2347.978648] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2347.980883] Lustre: Skipped 1 previous similar message [ 2352.294925] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2352.298053] Lustre: Skipped 3 previous similar messages [ 2360.938215] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2360.944430] Lustre: Skipped 8 previous similar messages [ 2377.519507] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2377.522727] Lustre: Skipped 15 previous similar messages [ 2409.801834] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2409.804973] Lustre: Skipped 30 previous similar messages [ 2458.331220] Lustre: 87775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2461.969332] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 15:23:56 (1787858636) [ 2463.871961] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2466.768356] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 15:24:01 (1787858641) [ 2473.444368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2473.446331] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2473.475373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2473.499968] Lustre: Skipped 1 previous similar message [ 2478.574156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2478.583245] Lustre: Skipped 1 previous similar message [ 2483.685782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2483.709118] Lustre: Skipped 1 previous similar message [ 2486.242234] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2486.595628] Lustre: server umount lustre-OST0000 complete [ 2492.387517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2492.399386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2492.419758] Lustre: Skipped 1 previous similar message [ 2492.428090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2493.926735] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2497.577511] Lustre: server umount lustre-MDT0000 complete [ 2499.043945] LustreError: 79653:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2499.077470] LustreError: 79653:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2502.173537] LustreError: 87777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858679 with bad export cookie 13593097265872565688 [ 2502.176794] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2502.221316] LustreError: 87777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2502.546818] Lustre: server umount lustre-MDT0001 complete [ 2515.280226] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 2519.021485] Key type lgssc unregistered [ 2519.420510] LNet: 89031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2519.426451] LNetError: 89031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2519.453443] LNet: Removed LNI 192.168.202.135@tcp [ 2520.281257] Key type .llcrypt unregistered [ 2520.283226] Key type ._llcrypt unregistered [ 2525.658416] Key type ._llcrypt registered [ 2525.666653] Key type .llcrypt registered [ 2525.786525] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2530.838378] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules [ 2532.652948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2532.727442] alg: No test for adler32 (adler32-zlib) [ 2533.840580] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 2534.154774] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 2535.897741] Key type lgssc registered [ 2537.746702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2539.056734] LustreError: 90247:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2560.867601] LustreError: 90247:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2561.697774] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 2574.490336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2574.500644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2576.394821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2584.264372] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2586.604894] LustreError: 91259:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2586.646374] LustreError: 91259:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2591.720426] LustreError: 91258:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.607716] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 15:26:11 (1787858771) [ 2596.838822] LustreError: 91259:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.954571] LustreError: 91258:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2602.075894] LustreError: 91290:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2602.090514] LustreError: 91290:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2602.097927] LustreError: 91290:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 26, retries 0, failed: rc = -5 [ 2602.581607] Lustre: server umount lustre-MDT0000 complete [ 2620.364409] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 2623.781537] Key type lgssc unregistered [ 2624.068419] LNet: 92786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2624.086374] LNetError: 92786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2624.105561] LNet: Removed LNI 192.168.202.135@tcp [ 2625.093782] Key type .llcrypt unregistered [ 2625.096420] Key type ._llcrypt unregistered [ 2646.542856] Key type ._llcrypt registered [ 2646.544889] Key type .llcrypt registered [ 2646.629426] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 2647.764974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2648.013189] alg: No test for adler32 (adler32-zlib) [ 2649.028222] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 2649.232413] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 2650.872183] Key type lgssc registered [ 2652.097180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2660.437260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2660.462264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.135445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2667.638091] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2672.618366] LustreError: 94007:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2672.645476] LustreError: 94007:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2677.734092] LustreError: 94006:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.270664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2679.539633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2684.744562] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2694.984579] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2702.817397] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2711.388581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2712.388153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2719.634963] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2729.282749] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2791.654208] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 15:29:26 (1787858966) [ 2793.711361] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2796.194337] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 15:29:30 (1787858970) [ 2801.633609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2801.649398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2801.667302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2804.708096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2804.713310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2804.722792] Lustre: Skipped 1 previous similar message [ 2804.730116] Lustre: Skipped 1 previous similar message [ 2807.700614] Lustre: server umount lustre-MDT0000 complete [ 2809.835335] LustreError: 94002:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2809.882582] LustreError: 94002:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2811.829132] LustreError: 94748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858988 with bad export cookie 5014082908138099953 [ 2811.835763] LustreError: 94748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2811.838291] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2812.132799] Lustre: server umount lustre-MDT0001 complete [ 2826.659825] Lustre: server umount lustre-OST0000 complete [ 2841.633530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2842.258646] LustreError: 98310:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2842.550808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2847.714062] LustreError: 98311:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2848.293554] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2857.955420] LustreError: 98310:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2857.968900] LustreError: 98310:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2858.453922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2859.269607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2864.698090] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2874.986873] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2882.784719] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2899.869182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2900.150514] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2900.155762] LustreError: 98298:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2900.175600] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2900.180848] LustreError: 100396:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2900.185513] LustreError: 100396:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2900.188850] LustreError: 100396:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2900.236281] Lustre: server umount lustre-OST0000 complete [ 2900.238478] LustreError: 100396:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2910.416110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2910.711489] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2910.723654] LustreError: 98300:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2910.794228] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2910.817680] LustreError: 100863:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2910.832668] LustreError: 100863:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2910.843136] LustreError: 100863:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2910.996528] Lustre: server umount lustre-OST0000 complete [ 2910.998624] LustreError: 100863:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2926.820485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2927.230564] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2927.250312] Lustre: Found index 0 for lustre-OST0000, updating log [ 2927.268249] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2927.422689] Lustre: lustre-OST0000: new disk, initializing [ 2927.429103] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2927.440241] Lustre: 101397:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2928.148541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2936.504572] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 2940.792545] Lustre: Failing over lustre-OST0000 [ 2941.167492] Lustre: server umount lustre-OST0000 complete [ 2947.554572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2947.557624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2947.568692] Lustre: Skipped 1 previous similar message [ 2947.583351] Lustre: Skipped 2 previous similar messages [ 2952.673744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2952.695800] Lustre: Skipped 1 previous similar message [ 2957.803590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2957.807892] Lustre: Skipped 1 previous similar message [ 2957.967433] Lustre: server umount lustre-MDT0000 complete [ 2958.816207] Lustre: 93319:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787859119/real 1787859119] req@00000000fcce7b72 x1874705856445696/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787859135 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2958.868312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2958.897686] LustreError: 99042:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2958.926185] LustreError: 99042:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2960.931353] Lustre: 93321:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787859121/real 1787859121] req@000000001d3d316e x1874705856445952/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787859137 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2963.336533] LustreError: 100416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859140 with bad export cookie 5014082908138102697 [ 2963.344091] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2963.370269] LustreError: 100416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2963.819586] Lustre: server umount lustre-MDT0001 complete [ 2975.881167] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 15:32:30 (1787859150) [ 2996.082792] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 3008.523455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3009.090093] LustreError: 103748:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3009.214522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3014.814141] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3024.683548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3031.099928] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3034.689455] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3043.964715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3044.731673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3044.741993] Lustre: Skipped 1 previous similar message [ 3044.796152] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3050.857843] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3052.949456] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3055.071366] Lustre: lustre-OST0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3055.074544] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3055.084212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 3066.983930] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3067.160850] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3067.284914] Lustre: lustre-OST0001: new disk, initializing [ 3067.288331] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3067.294719] Lustre: 106071:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3076.855901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3076.870346] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3076.971119] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3077.074382] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3087.467670] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3096.093182] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3101.380804] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.35@tcp (stopping) [ 3102.701601] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3102.716298] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3102.723301] Lustre: Skipped 2 previous similar messages [ 3107.418342] Lustre: server umount lustre-MDT0001 complete [ 3107.814788] LustreError: 106424:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3107.834627] LustreError: 106424:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [ 3120.540606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3120.691372] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3120.697678] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3120.753863] Lustre: 103738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3121.786667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3121.802658] Lustre: Skipped 1 previous similar message [ 3122.014505] LustreError: 107592:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3122.035339] LustreError: 107592:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3122.046637] LustreError: 103740:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3122.232239] LustreError: 107608:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.135@tcp: cfg command failed: rc = -17 [ 3122.239671] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3122.239671] [ 3127.287127] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3127.302559] Lustre: Skipped 1 previous similar message [ 3127.500169] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3140.322647] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 15:35:15 (1787859315) [ 3165.269425] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 3186.594960] Lustre: 109709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3211.839914] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3216.756231] Lustre: 110844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3311.586152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3311.590494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3311.607684] Lustre: Skipped 1 previous similar message [ 3311.611600] Lustre: Skipped 4 previous similar messages [ 3313.449165] Lustre: server umount lustre-MDT0000 complete [ 3316.706036] LustreError: 103749:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3316.731898] LustreError: 103749:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 11 previous similar messages [ 3317.698143] LustreError: 103729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859494 with bad export cookie 5014082908138103705 [ 3317.702799] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3317.716111] LustreError: 103729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3318.126217] Lustre: server umount lustre-MDT0001 complete [ 3333.968519] Lustre: server umount lustre-OST0000 complete [ 3337.701512] Lustre: 93321:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787859498/real 1787859498] req@ffff89c62fe392c0 x1874705856726016/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787859514 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3337.736544] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3337.744029] Lustre: Skipped 2 previous similar messages [ 3339.084571] Lustre: server umount lustre-OST0001 complete [ 3347.090347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3358.886516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3369.695458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3382.276484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3409.640328] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 3421.309531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3421.325862] Lustre: lustre-MDT0000: reset Object Index mappings [ 3421.730134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3427.383871] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3438.324220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3438.364267] Lustre: lustre-MDT0001: reset Object Index mappings [ 3445.648125] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3449.198210] Lustre: 115956:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3456.637488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3463.629721] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 3465.565746] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3466.723402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3476.125757] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3482.626235] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3482.629294] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3485.061487] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3495.051809] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3499.588908] Lustre: 117834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3561.142919] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 15:42:15 (1787859735) [ 3587.555424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3587.562447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3587.574280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3587.582782] Lustre: Skipped 3 previous similar messages [ 3593.199848] LustreError: 114808:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3593.235624] LustreError: 114808:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 12 previous similar messages [ 3595.023124] Lustre: server umount lustre-MDT0000 complete [ 3599.745641] LustreError: 114789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859776 with bad export cookie 5014082908138150794 [ 3599.752769] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3599.764114] LustreError: 114789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3606.506933] Lustre: server umount lustre-MDT0001 complete [ 3611.394596] LustreError: 118597:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000400:0xbb:0x0].0x0 (ffff89c607fa2700) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3617.795928] Lustre: server umount lustre-OST0000 complete [ 3629.065361] Lustre: server umount lustre-OST0001 complete [ 3637.700280] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 3651.064707] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 3702.854472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3703.059745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3703.115047] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3703.238594] Lustre: lustre-MDT0000: new disk, initializing [ 3703.370992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3703.375996] Lustre: Skipped 3 previous similar messages [ 3703.399286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3709.463753] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3718.387538] Lustre: server umount lustre-MDT0000 complete [ 3731.331764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3731.865595] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3731.917068] Lustre: 122123:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3731.945923] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3738.044916] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3744.019172] Lustre: server umount lustre-MDT0000 complete [ 3755.086738] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 15:45:29 (1787859929) [ 3757.495573] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3760.025816] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 15:45:34 (1787859934) [ 3784.527286] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 3796.197980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3802.921559] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3814.623861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3814.818090] Lustre: 124197:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3814.913581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3814.928968] Lustre: Skipped 1 previous similar message [ 3815.150392] Lustre: lustre-MDT0001: new disk, initializing [ 3815.410935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3815.423746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3821.488509] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3827.484454] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3836.760245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3837.236865] Lustre: lustre-OST0000: new disk, initializing [ 3837.240421] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3837.245893] Lustre: 125774:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3838.761956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3838.786977] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3838.964867] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3846.691986] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3861.369516] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3861.627254] Lustre: lustre-OST0001: new disk, initializing [ 3861.632917] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3861.644618] Lustre: 126645:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3863.205813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3863.229472] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3863.349846] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3869.493425] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 3881.559706] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3886.349618] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4307.309422] Lustre: 133064:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4307.317792] Lustre: 133064:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4307.325892] Lustre: 133064:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4307.335283] Lustre: 133064:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4307.343183] Lustre: 133064:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4307.355052] Lustre: 133064:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4608.499448] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:59:41 (1787860781) [ 4626.407131] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4626.408270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4626.447291] Lustre: Skipped 8 previous similar messages [ 4626.474114] Lustre: Skipped 11 previous similar messages [ 4631.976291] Lustre: server umount lustre-MDT0000 complete [ 4636.484209] LustreError: 124190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860813 with bad export cookie 5014082908138220521 [ 4636.487048] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4636.495168] LustreError: 124190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4636.643847] LustreError: 127451:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4636.651302] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4636.657129] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4636.657139] Lustre: Skipped 4 previous similar messages [ 4636.666757] LustreError: 127451:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 9 previous similar messages [ 4636.699950] Lustre: Skipped 1 previous similar message [ 4643.214805] Lustre: server umount lustre-MDT0001 complete [ 4654.257271] Lustre: server umount lustre-OST0000 complete [ 4664.708953] Lustre: server umount lustre-OST0001 complete [ 4672.345627] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 4683.453365] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 4724.997619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4725.298554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4725.420548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4725.551783] Lustre: lustre-MDT0000: new disk, initializing [ 4725.676942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4725.684603] Lustre: Skipped 5 previous similar messages [ 4725.712567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4731.769453] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4745.226169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4745.351623] Lustre: 136130:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4745.413593] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4745.417355] Lustre: Skipped 1 previous similar message [ 4745.527303] Lustre: lustre-MDT0001: new disk, initializing [ 4745.736719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4745.762331] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4752.564523] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4763.058210] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4770.447518] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4778.806673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4779.215868] Lustre: lustre-OST0000: new disk, initializing [ 4779.220067] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4779.245260] Lustre: 138101:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4780.931633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4780.942298] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4781.143037] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4788.133790] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 4799.972771] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4812.218864] Lustre: lustre-MDD0000: changelog on [ 4815.512859] Lustre: lustre-MDD0001: changelog on [ 6901.247737] Lustre: lustre-MDD0001: changelog off [ 6903.695145] Lustre: 139544:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6904.782885] Lustre: lustre-MDD0000: changelog off [ 6916.589346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6916.599525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6916.604026] LustreError: Skipped 1 previous similar message [ 6916.613358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6916.627365] Lustre: Skipped 1 previous similar message [ 6916.655224] Lustre: Skipped 4 previous similar messages [ 6918.844389] Lustre: server umount lustre-OST0000 complete [ 6926.819730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6926.821754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6926.845364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6928.988199] Lustre: server umount lustre-MDT0000 complete [ 6931.937919] LustreError: 136143:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6931.960429] LustreError: 136143:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 6934.337216] LustreError: 136123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863111 with bad export cookie 5014082908139044246 [ 6934.354348] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6934.368189] LustreError: 136123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6935.077845] Lustre: server umount lustre-MDT0001 complete [ 6950.241960] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 6953.394505] Key type lgssc unregistered [ 6953.823562] LNet: 147259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6953.833320] LNetError: 147259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6953.853235] LNet: Removed LNI 192.168.202.135@tcp [ 6954.984182] Key type .llcrypt unregistered [ 6954.985887] Key type ._llcrypt unregistered [ 6976.874304] Key type ._llcrypt registered [ 6976.876972] Key type .llcrypt registered [ 6977.008086] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 6997.492531] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 6998.428623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6998.440244] alg: No test for adler32 (adler32-zlib) [ 6999.661607] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 6999.913668] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 7001.600237] Key type lgssc registered [ 7002.991897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7046.970835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7047.013884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7048.362165] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7048.429555] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7048.564048] Lustre: lustre-MDT0000: new disk, initializing [ 7048.640172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7048.660233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7053.396357] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7067.976746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7068.215040] Lustre: 150477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7068.315811] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7068.323858] Lustre: Skipped 1 previous similar message [ 7068.450470] Lustre: lustre-MDT0001: new disk, initializing [ 7068.649031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7068.759782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7068.790901] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7075.084218] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7085.252599] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7091.664580] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7101.006748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7101.394872] Lustre: lustre-OST0000: new disk, initializing [ 7101.399578] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7101.417487] Lustre: 152447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7101.535547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7111.620557] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7111.828213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7111.855927] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7112.114540] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7123.134189] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7129.933885] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_module obdecho/obdecho [ 7131.910543] LustreError: 153452:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7131.923834] LustreError: 153452:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7134.014444] LustreError: 153548:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7148.003223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7148.031841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7148.040300] Lustre: Skipped 1 previous similar message [ 7152.059980] Lustre: server umount lustre-MDT0000 complete [ 7156.714267] LustreError: 150470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863333 with bad export cookie 5642121148121112984 [ 7156.716031] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7156.724852] LustreError: 150470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7157.075485] Lustre: server umount lustre-MDT0001 complete [ 7173.916930] Lustre: server umount lustre-OST0000 complete [ 7198.046612] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 7212.069937] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 7252.523037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7252.727447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7252.752609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7252.834579] Lustre: lustre-MDT0000: new disk, initializing [ 7252.910934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7252.959495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7257.238201] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7268.276975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7268.398506] Lustre: 157278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7268.487133] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7268.500041] Lustre: Skipped 1 previous similar message [ 7268.615652] Lustre: lustre-MDT0001: new disk, initializing [ 7268.666766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7268.718889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7268.728114] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7272.959762] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7283.172874] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7290.054313] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7297.490601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7297.801346] Lustre: lustre-OST0000: new disk, initializing [ 7297.805577] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7297.816113] Lustre: 159249:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7297.910234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7304.138210] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7305.784564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7305.792334] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7305.889387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7313.568381] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7318.329775] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7318.478093] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7323.257551] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7323.534184] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7326.692229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7326.717718] Lustre: Skipped 1 previous similar message [ 7326.718245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7326.729867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7326.738450] Lustre: Skipped 3 previous similar messages [ 7331.810836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7331.830806] Lustre: Skipped 1 previous similar message [ 7332.367824] Lustre: server umount lustre-OST0000 complete [ 7336.930811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7336.941838] Lustre: Skipped 1 previous similar message [ 7336.955951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7336.961217] Lustre: Skipped 1 previous similar message [ 7342.050651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7342.055727] Lustre: Skipped 1 previous similar message [ 7342.614880] Lustre: server umount lustre-MDT0000 complete [ 7346.499056] LustreError: 157272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863523 with bad export cookie 5642121148121113936 [ 7346.509016] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7346.511606] LustreError: 157272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7346.893829] Lustre: server umount lustre-MDT0001 complete [ 7351.426209] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 16:45:26 (1787863526) [ 7353.330612] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7356.127604] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 16:45:30 (1787863530) [ 7365.315159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7365.692492] LustreError: 161659:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7365.712023] LustreError: 161659:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 7365.765849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7371.216672] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7371.246357] LustreError: 161660:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7375.336069] LustreError: 161659:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7380.458194] LustreError: 161660:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7382.246280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7389.224341] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7397.340939] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7404.517192] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7413.008053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7413.488441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7413.499669] Lustre: Skipped 1 previous similar message [ 7420.528696] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7427.393723] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7444.451501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7444.456793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7444.468973] Lustre: Skipped 1 previous similar message [ 7444.494092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7450.440244] Lustre: server umount lustre-MDT0000 complete [ 7450.594757] LustreError: 161660:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7450.629429] LustreError: 161660:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7455.237077] LustreError: 161639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863632 with bad export cookie 5642121148121114825 [ 7455.260690] LustreError: 161639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7455.266911] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7455.719473] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7455.731301] Lustre: Skipped 2 previous similar messages [ 7458.985344] LustreError: 161655:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7459.012855] LustreError: 161655:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7460.834119] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7460.839942] Lustre: Skipped 6 previous similar messages [ 7461.824659] Lustre: server umount lustre-MDT0001 complete [ 7468.759612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7482.874779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7483.042515] LustreError: 165528:0:(llog.c:1655:llog_backup()) MGC192.168.202.135@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7483.048562] Lustre: 165528:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.135@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7485.473948] LustreError: 165533:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7485.488384] LustreError: 165533:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7519.916140] LustreError: 165552:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7519.929109] LustreError: 165552:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 27 previous similar messages [ 7543.776431] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7543.787532] Lustre: 165528:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.135@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7584.764421] LustreError: 165535:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7584.770883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7584.807405] LustreError: 165535:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 42 previous similar messages [ 7589.886272] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7592.297725] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7602.764873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7615.460610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7631.358847] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7631.370848] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7639.555820] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7641.662275] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7648.659373] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7651.286928] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7672.808253] Lustre: server umount lustre-MDT0000 complete [ 7677.051672] LustreError: 161640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863854 with bad export cookie 5642121148121121314 [ 7677.065379] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7677.081180] LustreError: 161640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7677.440253] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7677.737581] Lustre: server umount lustre-MDT0001 complete [ 7682.203220] Lustre: server umount lustre-OST0000 complete [ 7708.545936] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 7721.341918] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 7763.899111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7764.148304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7764.171141] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7764.246335] Lustre: lustre-MDT0000: new disk, initializing [ 7764.313663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7764.319559] Lustre: Skipped 1 previous similar message [ 7764.331701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7769.720951] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7780.703928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7780.890756] Lustre: 170634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7781.372178] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7785.924556] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7795.614546] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7801.085949] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7807.567944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7807.835859] Lustre: lustre-OST0000: new disk, initializing [ 7807.839102] Lustre: Skipped 1 previous similar message [ 7807.843380] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7807.850933] Lustre: Skipped 2 previous similar messages [ 7807.860096] Lustre: 172603:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7814.412245] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7814.733086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7814.756034] Lustre: Skipped 1 previous similar message [ 7814.777428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7814.892304] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7825.603631] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7832.609415] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7833.021691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7839.485477] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7839.771224] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7845.858856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7845.862885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7845.878067] LustreError: Skipped 1 previous similar message [ 7845.883537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7845.896259] Lustre: Skipped 2 previous similar messages [ 7845.927842] Lustre: Skipped 1 previous similar message [ 7848.534717] Lustre: server umount lustre-OST0000 complete [ 7853.026254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7867.363099] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7867.806552] Lustre: server umount lustre-MDT0000 complete [ 7870.435232] LustreError: 170647:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7870.455979] LustreError: 170647:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 36 previous similar messages [ 7872.140868] LustreError: 172602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864049 with bad export cookie 5642121148121138114 [ 7872.144496] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7872.151102] LustreError: 172602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7872.491389] Lustre: server umount lustre-MDT0001 complete [ 7883.101762] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 16:54:18 (1787864058) [ 7890.590946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7896.735717] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7907.886790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7914.426111] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7922.575490] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7929.237840] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7938.244891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7946.211824] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 7954.999699] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7979.652417] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 16:55:54 (1787864154) [ 7987.169421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7987.181770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7987.191213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7987.197101] Lustre: Skipped 3 previous similar messages [ 7987.218242] Lustre: Skipped 7 previous similar messages [ 7992.132607] Lustre: server umount lustre-OST0000 complete [ 8002.935935] Lustre: server umount lustre-MDT0000 complete [ 8006.181077] LustreError: 175002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864183 with bad export cookie 5642121148121138996 [ 8006.186492] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8006.187041] LustreError: 175002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8006.576960] Lustre: server umount lustre-MDT0001 complete [ 8021.007421] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 8025.590397] Key type lgssc unregistered [ 8025.924408] LNet: 178777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8025.935475] LNetError: 178777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8025.960494] LNet: Removed LNI 192.168.202.135@tcp [ 8027.340503] Key type .llcrypt unregistered [ 8027.344345] Key type ._llcrypt unregistered [ 8050.125449] Key type ._llcrypt registered [ 8050.128866] Key type .llcrypt registered [ 8050.266940] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 8051.553273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8051.652289] alg: No test for adler32 (adler32-zlib) [ 8052.798230] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 8053.081910] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 8054.744195] Key type lgssc registered [ 8056.119206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8065.112966] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8065.137504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8080.672480] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8085.792721] LustreError: 180021:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.135@tcp: failed processing log, type 4: rc = -110 [ 8090.912476] LustreError: 180021:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8090.936565] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8090.943692] LustreError: MGC192.168.202.135@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8090.952116] LustreError: 180021:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8090.959533] LustreError: 180021:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 8090.965572] LustreError: 180021:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8091.053176] Lustre: server umount lustre-OST0000 complete [ 8091.055238] LustreError: 180021:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8100.837498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8102.425089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8107.149459] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8112.628041] LustreError: 180521:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8112.647925] LustreError: 180521:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8117.074734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8117.605220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8123.028752] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8132.381693] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8140.197628] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8148.649272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8148.922870] LustreError: 182340:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8148.935627] Lustre: 182340:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.135@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8149.111954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8156.049925] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8164.952996] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8167.909937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8167.915576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8167.930134] Lustre: Skipped 2 previous similar messages [ 8167.935162] Lustre: Skipped 1 previous similar message [ 8173.028865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8173.038821] Lustre: Skipped 2 previous similar messages [ 8173.528682] Lustre: server umount lustre-MDT0000 complete [ 8178.057921] LustreError: 182367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864355 with bad export cookie 7880629955072817149 [ 8178.059206] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8178.068560] LustreError: 182367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8178.144915] LustreError: 181675:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8178.145163] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8178.183123] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8178.208266] LustreError: 181253:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000005af12aa0 x1874711522864768/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8178.641447] Lustre: server umount lustre-MDT0001 complete [ 8185.197638] Lustre: server umount lustre-OST0000 complete [ 8197.158696] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 8211.058265] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 8254.259984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8254.749285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8254.791683] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8254.877206] Lustre: lustre-MDT0000: new disk, initializing [ 8254.996312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8255.027808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8260.001775] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8272.181181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8272.373714] Lustre: 185880:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8272.456499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8272.460845] Lustre: Skipped 1 previous similar message [ 8272.534588] Lustre: lustre-MDT0001: new disk, initializing [ 8272.617501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8272.652867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8272.669748] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8277.639710] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8289.248094] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8296.375957] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8304.440370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8304.934735] Lustre: lustre-OST0000: new disk, initializing [ 8304.939668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8304.951466] Lustre: 187861:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8305.075334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8312.285370] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8313.381585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8313.405921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8313.548126] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8323.085862] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8328.777223] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8329.075073] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8334.360959] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8334.640140] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8339.425117] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8339.430246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8339.438140] LustreError: Skipped 1 previous similar message [ 8339.444261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8339.459518] Lustre: Skipped 1 previous similar message [ 8339.480996] Lustre: Skipped 1 previous similar message [ 8344.553155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8344.561381] Lustre: Skipped 1 previous similar message [ 8351.201829] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8351.575363] Lustre: server umount lustre-OST0000 complete [ 8359.907101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8359.908658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8359.919328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8359.948206] Lustre: Skipped 1 previous similar message [ 8359.959857] Lustre: Skipped 3 previous similar messages [ 8363.157502] Lustre: server umount lustre-MDT0000 complete [ 8365.026164] LustreError: 185891:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8365.060179] LustreError: 185891:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8367.620395] LustreError: 185871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864544 with bad export cookie 7880629955072817863 [ 8367.622457] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8367.632180] LustreError: 185871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8368.226599] Lustre: server umount lustre-MDT0001 complete [ 8378.845931] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 17:02:33 (1787864553) [ 8397.190511] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 8400.266401] Key type lgssc unregistered [ 8400.558945] LNet: 190811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8400.565786] LNetError: 190811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8400.582685] LNet: Removed LNI 192.168.202.135@tcp [ 8401.619082] Key type .llcrypt unregistered [ 8401.620763] Key type ._llcrypt unregistered [ 8420.683762] Key type ._llcrypt registered [ 8420.689975] Key type .llcrypt registered [ 8420.837661] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 8422.596041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8422.802451] alg: No test for adler32 (adler32-zlib) [ 8424.012924] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 8424.456111] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 8426.240178] Key type lgssc registered [ 8427.823858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8438.673151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8438.717458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8440.270934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8444.579491] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8450.533541] LustreError: 192029:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8450.564334] LustreError: 192029:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8453.774521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8454.067845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8458.269894] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8466.835910] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8474.058974] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8482.998343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8483.234994] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8483.238187] LustreError: 193897:0:(llog.c:1696:llog_backup()) MGC192.168.202.135@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8483.251480] Lustre: 193897:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.135@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8483.416659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8490.915378] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8502.016409] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8509.412207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8509.416227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8509.440738] Lustre: Skipped 2 previous similar messages [ 8509.452437] Lustre: Skipped 2 previous similar messages [ 8511.961994] Lustre: server umount lustre-MDT0000 complete [ 8514.532709] LustreError: 192024:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8514.546353] LustreError: 192024:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8516.041627] LustreError: 192010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864693 with bad export cookie 5252970099937145080 [ 8516.064350] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8516.494278] Lustre: server umount lustre-MDT0001 complete [ 8531.411989] Lustre: server umount lustre-OST0000 complete [ 8542.943406] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 8556.155538] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 8597.199814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8597.380686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8597.435202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8597.480873] Lustre: lustre-MDT0000: new disk, initializing [ 8597.562903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8597.577882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8602.004626] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8614.566613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8614.773993] Lustre: 197425:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8614.856899] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8614.862662] Lustre: Skipped 1 previous similar message [ 8614.872534] LustreError: 198207:0:(llog.c:1696:llog_backup()) MGC192.168.202.135@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8614.880559] LustreError: 198207:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8614.886313] Lustre: 198207:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.135@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8614.955582] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8614.960187] Lustre: Skipped 1 previous similar message [ 8615.098677] Lustre: lustre-MDT0001: new disk, initializing [ 8615.295828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8615.354993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8615.369394] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8619.608512] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8628.582824] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8634.448704] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8641.153428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8641.427174] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8641.432577] Lustre: Skipped 1 previous similar message [ 8641.436329] LustreError: 199369:0:(llog.c:1696:llog_backup()) MGC192.168.202.135@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8641.443349] LustreError: 199369:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8641.446984] Lustre: 199369:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.135@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8641.455829] Lustre: 199369:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8641.468753] Lustre: lustre-OST0000: new disk, initializing [ 8641.472767] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8641.477884] Lustre: 199393:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8641.549755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8647.229206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8647.236635] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8647.308321] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8647.719725] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8657.426450] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8662.110420] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8662.339643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8666.762600] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8666.942742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8672.740753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8672.745538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8672.757346] Lustre: Skipped 1 previous similar message [ 8672.769333] Lustre: Skipped 1 previous similar message [ 8675.026749] Lustre: server umount lustre-OST0000 complete [ 8681.953728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8681.962754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8681.990948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8685.381973] Lustre: server umount lustre-MDT0000 complete [ 8688.109313] LustreError: 198794:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8688.140048] LustreError: 198794:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8689.877441] LustreError: 198220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864866 with bad export cookie 5252970099937145801 [ 8689.887086] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8689.897371] LustreError: 198220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8690.498824] Lustre: server umount lustre-MDT0001 complete [ 8699.623935] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 17:07:54 (1787864874) [ 8719.211853] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 8730.185196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8730.641539] LustreError: 202519:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8730.717713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8735.715426] LustreError: 202520:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8735.779407] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8745.976482] LustreError: 202520:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8745.995119] LustreError: 202520:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8746.172664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8752.783501] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8756.517112] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8764.859380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8765.319163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8765.327516] Lustre: Skipped 1 previous similar message [ 8771.612833] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8781.929099] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8782.097334] Lustre: lustre-OST0001: new disk, initializing [ 8782.101355] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8782.107214] Lustre: 204828:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8789.440628] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8790.082192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8790.093818] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8790.220932] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8802.417519] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8810.887493] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8812.984531] Lustre: DEBUG MARKER: ADD OST3 [ 8818.028854] loop: module loaded [ 8818.127467] loop0: detected capacity change from 0 to 204800000 [ 8818.426301] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8818.451786] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8818.466436] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8824.609657] loop0: detected capacity change from 0 to 204800000 [ 8825.754866] Lustre: DEBUG MARKER: STOP OST3 [ 8830.797362] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8831.021216] Lustre: lustre-OST0003: new disk, initializing [ 8831.030563] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8831.060555] Lustre: 206827:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8831.229040] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8831.243296] Lustre: Skipped 1 previous similar message [ 8832.988192] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8833.102133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8833.111031] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8833.164076] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8835.041089] LustreError: 207018:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8835.144353] Lustre: Failing over lustre-OST0003 [ 8835.293384] Lustre: server umount lustre-OST0003 complete [ 8837.788674] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8838.131086] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8838.136898] LustreError: 204005:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8838.151879] Lustre: Skipped 2 previous similar messages [ 8838.172851] LustreError: 204005:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8847.228916] Lustre: DEBUG MARKER: START OST3 again [ 8853.925276] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8854.471620] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8855.531626] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8856.334274] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8856.336718] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8860.320818] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 8880.833123] Lustre: Failing over lustre-OST0003 [ 8880.903424] Lustre: server umount lustre-OST0003 complete [ 8882.147036] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8882.147658] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8882.155264] LustreError: Skipped 1 previous similar message [ 8882.157383] LustreError: 204531:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8882.157394] LustreError: 204531:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 10 previous similar messages [ 8882.203324] Lustre: Skipped 1 previous similar message [ 8900.578497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8900.582704] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8900.585991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8900.587311] Lustre: Skipped 2 previous similar messages [ 8900.607231] Lustre: Skipped 3 previous similar messages [ 8905.702694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8905.712467] Lustre: Skipped 6 previous similar messages [ 8912.353427] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8912.568727] Lustre: server umount lustre-MDT0000 complete [ 8916.341656] LustreError: 202501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865093 with bad export cookie 5252970099937146683 [ 8916.349565] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8916.357442] LustreError: 202501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8921.059027] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8921.066617] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8921.080441] Lustre: Skipped 1 previous similar message [ 8921.102524] Lustre: Skipped 5 previous similar messages [ 8922.737083] Lustre: server umount lustre-MDT0001 complete [ 8926.646971] Lustre: server umount lustre-OST0000 complete [ 8930.656901] Lustre: server umount lustre-OST0001 complete [ 8937.591764] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 8946.776915] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 8986.323321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8986.462772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8986.486261] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8986.534951] Lustre: lustre-MDT0000: new disk, initializing [ 8986.585919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8986.596044] Lustre: Skipped 1 previous similar message [ 8986.614156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8991.095797] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9001.552423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9001.732901] Lustre: 212297:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9002.115789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9006.654680] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9016.948746] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9022.970031] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9030.080452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9030.253827] Lustre: lustre-OST0000: new disk, initializing [ 9030.256570] Lustre: Skipped 1 previous similar message [ 9030.262382] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9030.268079] Lustre: Skipped 2 previous similar messages [ 9030.273677] Lustre: 214263:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9031.628698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9031.641970] Lustre: Skipped 1 previous similar message [ 9031.654325] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9031.794722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9036.076802] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9044.932828] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9048.627838] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9048.789940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9053.334227] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9053.501378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9057.256618] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9057.257310] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9057.267053] LustreError: Skipped 1 previous similar message [ 9057.278158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9057.283396] Lustre: Skipped 1 previous similar message [ 9057.300649] Lustre: Skipped 1 previous similar message [ 9061.840574] Lustre: server umount lustre-OST0000 complete [ 9071.644991] Lustre: server umount lustre-MDT0000 complete [ 9072.610771] LustreError: 212309:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9072.633132] LustreError: 212309:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 18 previous similar messages [ 9075.158260] LustreError: 212290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865252 with bad export cookie 5252970099937148979 [ 9075.178627] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9075.507498] Lustre: server umount lustre-MDT0001 complete [ 9079.163123] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 17:14:14 (1787865254) [ 9095.800582] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 9107.159488] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 9148.482818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9148.718430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9148.751597] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9148.850412] Lustre: lustre-MDT0000: new disk, initializing [ 9148.961450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9153.496638] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9163.884354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9164.012702] Lustre: 218873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9164.344028] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9169.575763] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9181.271924] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9188.621980] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9196.428438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9196.862774] Lustre: 220845:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9198.786573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9198.952941] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9204.625954] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9216.919756] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9224.281874] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9224.678215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9232.575196] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9232.911234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9240.049577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9240.062885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9240.065138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9240.065147] Lustre: Skipped 2 previous similar messages [ 9240.065173] Lustre: Skipped 3 previous similar messages [ 9241.742822] Lustre: server umount lustre-OST0000 complete [ 9252.371584] Lustre: server umount lustre-MDT0000 complete [ 9255.405857] LustreError: 218886:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9255.439878] LustreError: 218886:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9256.565417] LustreError: 218867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865433 with bad export cookie 5252970099937149861 [ 9256.566595] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9256.584601] LustreError: 218867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9257.042805] Lustre: server umount lustre-MDT0001 complete [ 9267.562023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9268.253451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9268.259539] Lustre: Skipped 5 previous similar messages [ 9273.659849] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9284.144488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9289.747242] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9297.433278] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9305.682507] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9314.694991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9321.328254] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9329.166081] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9354.237593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9354.247744] Lustre: Skipped 5 previous similar messages [ 9357.711569] Lustre: server umount lustre-OST0000 complete [ 9368.292737] Lustre: server umount lustre-MDT0000 complete [ 9372.416262] LustreError: 223043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865549 with bad export cookie 5252970099937150736 [ 9372.419515] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9372.430058] LustreError: 223043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9372.805111] Lustre: server umount lustre-MDT0001 complete [ 9468.974829] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [ 9471.763598] Key type lgssc unregistered [ 9472.124931] LNet: 226753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9472.146271] LNetError: 226753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9473.192210] LNet: Removed LNI 192.168.202.135@tcp [ 9474.180329] Key type .llcrypt unregistered [ 9474.182807] Key type ._llcrypt unregistered [ 9503.606569] Key type ._llcrypt registered [ 9503.608121] Key type .llcrypt registered [ 9503.708373] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 9523.230885] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 9525.015194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9525.111366] alg: No test for adler32 (adler32-zlib) [ 9526.309656] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 9526.683485] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [ 9528.376554] Key type lgssc registered [ 9530.019354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9572.070669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9572.112559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9573.544671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9573.588544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9573.719720] Lustre: lustre-MDT0000: new disk, initializing [ 9573.896056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9573.916865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9579.981768] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9594.126336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9594.267259] Lustre: 230116:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9594.343867] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9594.348313] Lustre: Skipped 1 previous similar message [ 9594.418938] Lustre: lustre-MDT0001: new disk, initializing [ 9594.574528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9594.624538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9594.646780] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9600.063498] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9611.159661] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9617.337477] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9624.948558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9625.221321] Lustre: lustre-OST0000: new disk, initializing [ 9625.226445] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9625.232664] Lustre: 232086:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9625.311379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9633.400790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9633.406482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9633.597640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9633.818104] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9645.153450] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9648.643513] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9649.004332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9653.123201] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9653.458394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9659.370690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9659.373418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9659.378560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9659.381208] LustreError: Skipped 1 previous similar message [ 9659.412363] Lustre: Skipped 1 previous similar message [ 9661.777941] Lustre: server umount lustre-OST0000 complete [ 9666.529953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9666.539940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9666.561149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9666.570920] Lustre: Skipped 1 previous similar message [ 9669.616762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9669.621790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9674.721223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9674.728153] Lustre: Skipped 2 previous similar messages [ 9679.854519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9679.863848] Lustre: Skipped 1 previous similar message [ 9680.353476] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9680.506282] Lustre: server umount lustre-MDT0000 complete [ 9684.937142] LustreError: 230107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865861 with bad export cookie 10239479155864709574 [ 9684.940548] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9684.963420] LustreError: 230107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9685.469499] Lustre: server umount lustre-MDT0001 complete [ 9708.548630] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 9719.102519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9719.505368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9724.300266] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9724.898322] LustreError: 235021:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9724.918310] LustreError: 235021:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 9730.029407] LustreError: 235022:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9734.265293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9734.837919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9739.218609] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9742.628863] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9749.135096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9749.430842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9755.901789] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9764.762826] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9764.932617] Lustre: lustre-OST0001: new disk, initializing [ 9764.936260] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9764.941690] Lustre: 237328:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9771.497179] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9773.164764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9773.171913] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9773.272939] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9784.588193] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9791.685818] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9812.699497] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 17:26:26 (1787865986) [ 9816.547530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9816.567889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9816.591464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9830.369374] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9830.483138] Lustre: server umount lustre-MDT0000 complete [ 9834.994464] LustreError: 235017:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9835.019446] LustreError: 235017:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 9835.341821] LustreError: 235001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866012 with bad export cookie 10239479155864710456 [ 9835.352131] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9835.368400] LustreError: 235001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9836.081214] Lustre: server umount lustre-MDT0001 complete [ 9851.175202] Lustre: server umount lustre-OST0000 complete [ 9864.469789] Lustre: server umount lustre-OST0001 complete [ 9872.075098] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [ 9888.987789] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [ 9931.314316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9931.470680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9931.495409] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9931.593485] Lustre: lustre-MDT0000: new disk, initializing [ 9931.658960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9931.663633] Lustre: Skipped 1 previous similar message [ 9931.683436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9936.880359] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9948.159333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9948.279874] Lustre: 241313:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9948.328861] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9948.332608] Lustre: Skipped 1 previous similar message [ 9948.386847] Lustre: lustre-MDT0001: new disk, initializing [ 9948.481941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9948.496655] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9953.763385] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9963.347450] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9969.250691] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9977.089888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9977.366278] Lustre: lustre-OST0000: new disk, initializing [ 9977.371203] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9977.380694] Lustre: 243283:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9977.467669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9977.471333] Lustre: Skipped 1 previous similar message [ 9978.767921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9978.776641] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9978.896585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9983.543801] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [ 9993.528529] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9999.121319] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9999.303710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10004.359299] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10004.576684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10009.581513] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10009.585648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10009.595655] LustreError: Skipped 1 previous similar message [10009.597602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10009.597609] Lustre: Skipped 12 previous similar messages [10009.632317] Lustre: Skipped 4 previous similar messages [10012.636987] Lustre: server umount lustre-OST0000 complete [10019.822101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10019.839553] Lustre: Skipped 1 previous similar message [10023.372788] Lustre: server umount lustre-MDT0000 complete [10024.930893] LustreError: 242190:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10024.975033] LustreError: 242190:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10027.883202] LustreError: 241305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866204 with bad export cookie 10239479155864712591 [10027.889501] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10027.903451] LustreError: 241305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10028.552829] Lustre: server umount lustre-MDT0001 complete [10038.495262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10039.148272] LustreError: 245498:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10044.458242] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10049.506972] LustreError: 245499:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10049.541297] LustreError: 245499:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10054.816911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10055.273414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10055.285841] Lustre: Skipped 1 previous similar message [10060.287891] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10071.185473] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10079.148418] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10085.479584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10092.470223] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10100.358363] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10247.653515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10247.682093] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10247.684308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10247.705109] Lustre: Skipped 4 previous similar messages [10252.610077] Lustre: server umount lustre-OST0000 complete [10262.874280] Lustre: server umount lustre-MDT0000 complete [10263.011378] LustreError: 245493:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10263.022456] LustreError: 245493:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10267.344081] LustreError: 245478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866444 with bad export cookie 10239479155864713473 [10267.355441] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10267.357140] LustreError: 245478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10267.879981] Lustre: server umount lustre-MDT0001 complete [10370.454207] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [10372.548744] Key type lgssc unregistered [10372.782546] LNet: 249106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10372.789407] LNetError: 249106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10372.802639] LNet: Removed LNI 192.168.202.135@tcp [10373.600165] Key type .llcrypt unregistered [10373.605174] Key type ._llcrypt unregistered [10399.133218] Key type ._llcrypt registered [10399.135140] Key type .llcrypt registered [10399.240994] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [10416.228581] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [10417.658770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10417.730223] alg: No test for adler32 (adler32-zlib) [10418.834288] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [10419.121914] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [10420.872616] Key type lgssc registered [10421.957513] Lustre: Echo OBD driver; http://www.lustre.org/ [10471.896741] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [10482.647350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10482.671767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10483.975891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10484.014495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10484.122832] Lustre: lustre-MDT0000: new disk, initializing [10484.205303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10484.219668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10490.265353] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10502.063144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10502.233385] Lustre: 253196:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10502.311256] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10502.314781] Lustre: Skipped 1 previous similar message [10502.370534] Lustre: lustre-MDT0001: new disk, initializing [10502.440872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10502.469730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10502.479667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10507.021884] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10512.342691] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10520.627560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10520.958764] Lustre: lustre-OST0000: new disk, initializing [10520.964834] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10520.971900] Lustre: 254830:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10521.089367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10523.271055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10523.290160] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10523.451394] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10529.369501] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10541.637906] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10541.848308] Lustre: lustre-OST0001: new disk, initializing [10541.851418] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10541.856270] Lustre: 255703:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10541.917325] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10547.348299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10547.361224] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10547.508168] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10548.564800] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10561.319264] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10568.664743] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10579.427989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10579.433946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10579.443460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10583.529420] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10583.539097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10583.546699] Lustre: Skipped 1 previous similar message [10583.560625] Lustre: Skipped 3 previous similar messages [10588.651411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10588.657285] Lustre: Skipped 3 previous similar messages [10593.248184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10593.508945] Lustre: server umount lustre-MDT0000 complete [10597.171040] LustreError: 253990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866774 with bad export cookie 16483423921563456478 [10597.177460] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10597.191575] LustreError: 253990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10597.560155] Lustre: server umount lustre-MDT0001 complete [10611.224183] Lustre: server umount lustre-OST0000 complete [10615.008921] Lustre: 250682:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787866775/real 1787866775] req@ffff89c62eac21c0 x1874714003847168/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787866791 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10615.055559] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10615.073626] Lustre: Skipped 1 previous similar message [10615.680920] Lustre: server umount lustre-OST0001 complete [10631.130289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10639.805718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10648.457942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10659.802641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10685.165811] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [10696.350169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10696.394939] Lustre: lustre-MDT0000: reset Object Index mappings [10696.860190] LustreError: 260712:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10696.872489] LustreError: 260712:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [10696.934342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10701.758679] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10702.322733] LustreError: 260713:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10706.433185] LustreError: 260712:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10711.529386] LustreError: 260713:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10712.331111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10712.354132] Lustre: lustre-MDT0001: reset Object Index mappings [10712.682780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10718.166406] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10721.839652] Lustre: 261854:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10730.532272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10731.105963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10737.275209] LustreError: 262208:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10739.303348] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10741.389315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10746.855566] LustreError: 262442:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10746.893080] LustreError: 262442:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [10749.329830] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10755.075320] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10756.502443] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10764.949537] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10768.971450] Lustre: 263726:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10773.985526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10773.995347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10774.021883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10776.434659] Lustre: server umount lustre-MDT0000 complete [10780.225103] LustreError: 260693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866957 with bad export cookie 16483423921563463268 [10780.233977] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10780.241637] LustreError: 260693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10780.717655] Lustre: server umount lustre-MDT0001 complete [10795.054373] Lustre: server umount lustre-OST0000 complete [10797.026241] Lustre: 250681:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787866957/real 1787866957] req@ffff89c61837f840 x1874714003898496/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787866973 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10797.065905] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10797.095795] Lustre: Skipped 3 previous similar messages [10798.773418] Lustre: server umount lustre-OST0001 complete [10818.529613] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 17:43:13 (1787866993) [10835.980583] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [10845.467291] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [10886.055468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10886.394654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10886.457936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10886.555206] Lustre: lustre-MDT0000: new disk, initializing [10886.680981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10886.683885] Lustre: Skipped 1 previous similar message [10886.694435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10890.815840] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10901.694630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10901.819052] Lustre: 267983:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10901.876276] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10901.879746] Lustre: Skipped 1 previous similar message [10901.936159] Lustre: lustre-MDT0001: new disk, initializing [10902.043820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10902.060238] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10906.167955] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10916.223138] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10922.887040] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10931.890683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10932.140789] Lustre: lustre-OST0000: new disk, initializing [10932.144410] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10932.150514] Lustre: 269951:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10934.141803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10934.158030] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10934.315854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10939.138244] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [10950.109845] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10955.810311] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10956.063213] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10962.450362] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10962.786393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10970.086747] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10970.090033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10970.097257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10970.115520] Lustre: Skipped 1 previous similar message [10970.125375] Lustre: Skipped 5 previous similar messages [10979.296430] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10979.573291] Lustre: server umount lustre-OST0000 complete [10983.905244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10983.913851] LustreError: Skipped 1 previous similar message [10983.919906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10983.927425] Lustre: Skipped 2 previous similar messages [10989.986699] Lustre: server umount lustre-MDT0000 complete [10990.567181] LustreError: 267992:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10990.591939] LustreError: 267992:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10993.667607] LustreError: 267975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867170 with bad export cookie 16483423921563465361 [10993.672495] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10993.674663] LustreError: 267975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10993.922762] Lustre: server umount lustre-MDT0001 complete [11016.069171] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [11029.081112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11029.488444] LustreError: 272887:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11029.531951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11029.536862] Lustre: Skipped 2 previous similar messages [11035.283979] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11044.806838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11049.805622] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11052.906942] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11059.806587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11067.375716] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11077.769428] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11078.221484] Lustre: lustre-OST0001: new disk, initializing [11078.241967] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11078.250949] Lustre: 275192:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11079.887367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11079.907833] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11080.037864] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11087.382521] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11098.898886] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11102.663940] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11109.132230] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11109.475768] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11121.130700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11121.133137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11121.154351] Lustre: Skipped 5 previous similar messages [11121.164708] Lustre: Skipped 5 previous similar messages [11124.142484] Lustre: server umount lustre-MDT0000 complete [11126.253649] LustreError: 272888:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11126.277498] LustreError: 272888:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [11128.324462] LustreError: 272867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867305 with bad export cookie 16483423921563466243 [11128.333068] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11128.339461] LustreError: 272867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11128.625615] Lustre: server umount lustre-MDT0001 complete [11143.651710] Lustre: server umount lustre-OST0000 complete [11147.360118] Lustre: 250682:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787867308/real 1787867308] req@ffff89c63c705e00 x1874714004007296/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787867324 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11148.954929] Lustre: server umount lustre-OST0001 complete [11168.660594] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [11179.192066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11179.888645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11179.893694] Lustre: Skipped 3 previous similar messages [11184.380995] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11193.933876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11199.782875] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11204.232916] Lustre: 279392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11213.319202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11221.904500] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11224.090339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11230.766287] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11236.374430] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11238.186498] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11246.813698] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11250.723706] Lustre: 281271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11256.403673] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11256.757656] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11269.946184] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 17:50:43 (1787867443) [11286.498864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11286.506940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11286.524059] Lustre: Skipped 1 previous similar message [11286.533156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11292.646089] LustreError: 278247:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11292.662537] LustreError: 278247:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 12 previous similar messages [11292.728045] Lustre: server umount lustre-MDT0000 complete [11296.836317] LustreError: 278234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867473 with bad export cookie 16483423921563472585 [11296.850761] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11303.376555] Lustre: server umount lustre-MDT0001 complete [11322.339213] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11322.823285] Lustre: server umount lustre-OST0000 complete [11328.044717] Lustre: server umount lustre-OST0001 complete [11337.593378] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [11350.049193] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [11406.774766] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [11418.808845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11419.052839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11419.089685] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11419.261334] Lustre: lustre-MDT0000: new disk, initializing [11419.530403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11425.956118] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11440.619538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11440.731240] Lustre: 285519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11440.918550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11440.924000] Lustre: Skipped 4 previous similar messages [11440.989233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11448.682270] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11455.887018] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11465.485139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11465.731468] Lustre: 287157:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11467.391721] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11474.320243] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11487.517141] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11488.015063] Lustre: 288027:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11489.959976] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11489.977467] Lustre: Skipped 1 previous similar message [11490.154452] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11497.611575] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11509.701291] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11532.726136] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 17:55:07 (1787867707) [11536.361791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11536.364222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11536.382864] Lustre: Skipped 6 previous similar messages [11536.395938] Lustre: Skipped 9 previous similar messages [11537.300243] Lustre: server umount lustre-MDT0000 complete [11541.324688] LustreError: 285512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867718 with bad export cookie 16483423921563480453 [11541.338720] LustreError: 285512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11541.346539] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11541.914809] Lustre: server umount lustre-MDT0001 complete [11548.668759] Lustre: server umount lustre-OST0000 complete [11553.085603] Lustre: server umount lustre-OST0001 complete [11563.808298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11564.446197] LustreError: 290275:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11564.471344] LustreError: 290275:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [11571.152440] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11581.691489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11587.465861] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11597.481925] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11605.244591] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11614.138896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11621.582955] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11630.630848] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11655.622392] loop1: detected capacity change from 0 to 204800000 [11656.041376] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11656.051189] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11656.061284] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11662.523861] loop1: detected capacity change from 0 to 204800000 [11670.391862] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11670.623938] LustreError: 290265:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs1184: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11670.655510] LustreError: 290265:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs1184 [11670.669239] LustreError: 290265:0:(mgs_handler.c:517:mgs_target_reg()) fs1184-MDT0001: Error checking target index: rc = -13 [11670.705397] LustreError: fs1184-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11670.722055] LustreError: 293997:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11670.752551] LustreError: 293997:0:(tgt_mount.c:2129:server_put_super()) no obd fs1184-MDT0001 [11670.764221] LustreError: 293997:0:(tgt_mount.c:116:server_deregister_mount()) fs1184-MDT0001 not registered [11670.856710] Lustre: server umount fs1184-MDT0001 complete [11670.866553] LustreError: 293997:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11683.466746] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11683.763873] Lustre: Setting parameter fs1184-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs1184-MDT0001 [11683.785749] Lustre: Skipped 1 previous similar message [11683.879220] Lustre: srv-fs1184-MDT0001: No data found on store. Initialize space. [11683.884591] Lustre: Skipped 4 previous similar messages [11684.052954] Lustre: fs1184-MDT0001: new disk, initializing [11684.058617] Lustre: Skipped 3 previous similar messages [11684.179743] Lustre: srv-fs1184-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11684.190746] Lustre: 294466:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1184-MDT0001-mdtlov: can't create new llog: rc = -115 [11684.202480] LustreError: 294466:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1184-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11690.914734] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11701.853262] Lustre: server umount fs1184-MDT0001 complete [11710.972387] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11719.365458] loop1: detected capacity change from 0 to 204800000 [11726.570284] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11727.076391] Lustre: 296035:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1184-MDT0001-mdtlov: can't create new llog: rc = -115 [11727.092462] LustreError: 296035:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1184-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11731.518215] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11738.690857] Lustre: server umount fs1184-MDT0001 complete [11754.210585] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.35@tcp (stopping) [11754.220413] Lustre: Skipped 4 previous similar messages [11756.514787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11756.521280] LustreError: Skipped 1 previous similar message [11756.527455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11756.541961] Lustre: Skipped 4 previous similar messages [11758.779838] Lustre: server umount lustre-MDT0000 complete [11769.076436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11769.358806] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11774.725503] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11774.966876] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11784.248520] loop1: detected capacity change from 0 to 204800000 [11791.032768] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11791.559654] Lustre: 298484:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1184-MDT0001-mdtlov: can't create new llog: rc = -115 [11791.570784] LustreError: 298484:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1184-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11797.862684] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11804.243436] Lustre: server umount fs1184-MDT0001 complete [11833.841842] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11838.798140] Lustre: server umount lustre-OST0000 complete [11850.002805] Lustre: server umount lustre-MDT0000 complete [11855.196404] LustreError: 290256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868032 with bad export cookie 16483423921563483141 [11855.204450] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11855.213712] LustreError: 290256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11856.226636] Lustre: server umount lustre-MDT0001 complete [11870.277399] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [11874.011871] Key type lgssc unregistered [11874.610270] LNet: 300835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11874.627296] LNetError: 300835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11874.665713] LNet: Removed LNI 192.168.202.135@tcp [11876.062221] Key type .llcrypt unregistered [11876.067119] Key type ._llcrypt unregistered [11901.484463] Key type ._llcrypt registered [11901.489789] Key type .llcrypt registered [11901.652534] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [11902.965651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11903.071953] alg: No test for adler32 (adler32-zlib) [11904.444551] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [11904.792820] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [11906.520502] Key type lgssc registered [11908.156814] Lustre: Echo OBD driver; http://www.lustre.org/ [11917.566429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11917.603622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11919.301820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11925.337257] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11929.589248] LustreError: 302119:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11929.610213] LustreError: 302119:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11934.694128] LustreError: 302120:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11936.500613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11936.999437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11942.899424] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11951.597329] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11958.776471] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11967.190211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11967.826311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11975.994144] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [11981.160716] LustreError: 303955:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11985.425888] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11986.428456] LustreError: 303954:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11991.220479] LustreError: 303954:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11991.242838] LustreError: 303954:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11999.721096] LustreError: 303954:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11999.728160] LustreError: 303954:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [12011.054335] loop1: detected capacity change from 0 to 204800000 [12011.339442] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12011.363742] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12011.379239] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12016.804856] LustreError: 303956:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12016.821548] LustreError: 303956:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 9 previous similar messages [12017.710253] loop1: detected capacity change from 0 to 204800000 [12023.924787] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12024.125357] Lustre: fs138-OST0000: new disk, initializing [12024.131115] Lustre: srv-fs138-OST0000: No data found on store. Initialize space. [12024.138817] Lustre: 305859:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12024.234995] Lustre: fs138-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12032.840212] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12042.790685] Lustre: server umount fs138-OST0000 complete [12050.919593] LustreError: 303956:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12050.949597] LustreError: 303956:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 19 previous similar messages [12058.245643] loop1: detected capacity change from 0 to 204800000 [12061.161672] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12064.776373] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12065.276878] Lustre: fs138-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12066.288341] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12072.427549] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12082.555424] loop2: detected capacity change from 0 to 204800000 [12082.885233] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12082.920813] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12082.956833] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12089.040984] loop2: detected capacity change from 0 to 204800000 [12095.590973] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12095.709321] LustreError: fs138-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12095.731912] LustreError: fs138-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12095.743527] LustreError: 308900:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [12095.755524] LustreError: 308900:0:(tgt_mount.c:2129:server_put_super()) no obd fs138-OST0001 [12095.765194] LustreError: 308900:0:(tgt_mount.c:116:server_deregister_mount()) fs138-OST0001 not registered [12095.807627] Lustre: server umount fs138-OST0001 complete [12095.813720] LustreError: 308900:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12107.980597] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12108.172332] Lustre: fs138-OST0001: new disk, initializing [12108.177628] Lustre: srv-fs138-OST0001: No data found on store. Initialize space. [12108.185610] Lustre: 309451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12108.307032] Lustre: fs138-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12115.527503] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12117.481656] LustreError: 303955:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12117.507753] LustreError: 303955:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 38 previous similar messages [12125.516779] Lustre: server umount fs138-OST0000 complete [12138.124577] Lustre: server umount fs138-OST0001 complete [12155.665143] loop1: detected capacity change from 0 to 204800000 [12163.318777] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12163.627979] Lustre: fs138-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12172.045864] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12181.494333] loop2: detected capacity change from 0 to 204800000 [12188.023226] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12196.299048] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12206.803857] Lustre: server umount fs138-OST0000 complete [12214.781270] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12216.902493] Lustre: server umount fs138-OST0001 complete [12219.889857] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12232.163719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12232.176423] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12232.193922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12233.188790] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12233.200543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12233.236680] Lustre: Skipped 1 previous similar message [12238.698415] Lustre: server umount lustre-OST0000 complete [12243.432585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12243.434539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12243.458664] Lustre: Skipped 1 previous similar message [12243.471869] Lustre: Skipped 1 previous similar message [12248.545623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12248.552284] Lustre: Skipped 1 previous similar message [12249.809070] Lustre: server umount lustre-MDT0000 complete [12253.666324] LustreError: 302120:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12253.699991] LustreError: 302120:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 73 previous similar messages [12255.622534] LustreError: 302099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868432 with bad export cookie 3251073023223865999 [12255.638377] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12256.197447] Lustre: server umount lustre-MDT0001 complete [12270.803920] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [12274.171029] Key type lgssc unregistered [12274.480481] LNet: 315381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12274.492462] LNetError: 315381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12274.534716] LNet: Removed LNI 192.168.202.135@tcp [12275.469159] Key type .llcrypt unregistered [12275.471710] Key type ._llcrypt unregistered [12313.807386] Key type ._llcrypt registered [12313.811846] Key type .llcrypt registered [12313.911192] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [12332.053601] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [12333.364580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12333.404042] alg: No test for adler32 (adler32-zlib) [12334.466750] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [12334.715106] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [12336.416304] Key type lgssc registered [12337.850863] Lustre: Echo OBD driver; http://www.lustre.org/ [12383.689613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12383.715774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12385.025703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12385.071351] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12385.225554] Lustre: lustre-MDT0000: new disk, initializing [12385.377979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12385.427476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12391.742765] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12404.447224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12404.551736] Lustre: 319246:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12404.604475] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12404.607698] Lustre: Skipped 1 previous similar message [12404.714376] Lustre: lustre-MDT0001: new disk, initializing [12404.859214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12404.921711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12404.925606] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12411.399612] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12424.239514] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12430.105375] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12439.078173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12439.474811] Lustre: lustre-OST0000: new disk, initializing [12439.487979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12439.508714] Lustre: 321215:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12439.663499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12447.503028] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12448.367190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12448.383456] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12448.518635] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12458.720518] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12461.198832] Lustre: Mounted lustre-client - version 2.17.57_81_g80478ec [12462.441583] Lustre: Unmounted lustre-client [12474.337812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12474.340833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12474.356164] LustreError: Skipped 1 previous similar message [12474.366261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12474.377571] Lustre: Skipped 1 previous similar message [12474.395810] Lustre: Skipped 1 previous similar message [12476.903097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12476.922339] Lustre: Skipped 1 previous similar message [12478.429987] Lustre: server umount lustre-OST0000 complete [12486.639861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12486.649216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12486.664923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12489.074809] Lustre: server umount lustre-MDT0000 complete [12493.419794] LustreError: 319239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868670 with bad export cookie 8834629206124249883 [12493.423650] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12493.427424] LustreError: 319239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12494.014814] Lustre: server umount lustre-MDT0001 complete [12506.114770] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [12509.063992] Key type lgssc unregistered [12516.704540] LNet: 323380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12516.754163] LNetError: 323380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12592.550919] LNet: Removed LNI 192.168.202.135@tcp [12593.551869] Key type .llcrypt unregistered [12593.553472] Key type ._llcrypt unregistered [12613.687302] Key type ._llcrypt registered [12613.690453] Key type .llcrypt registered [12613.817778] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [12633.800688] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [12635.178157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12635.327068] alg: No test for adler32 (adler32-zlib) [12636.811069] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [12637.242314] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [12638.944280] Key type lgssc registered [12640.732871] Lustre: Echo OBD driver; http://www.lustre.org/ [12673.860288] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 18:14:08 (1787868848) [12681.355919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12681.392362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12682.717284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12682.793413] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12682.906947] Lustre: lustre-MDT0000: new disk, initializing [12683.004685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12683.015973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12688.393962] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12701.212956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12701.366162] Lustre: 326603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12701.443798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12701.446950] Lustre: Skipped 1 previous similar message [12701.514648] Lustre: lustre-MDT0001: new disk, initializing [12701.632299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12701.676438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12701.693091] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12708.669663] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12719.949082] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12726.435091] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12734.105807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12734.502692] Lustre: lustre-OST0000: new disk, initializing [12734.509769] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12734.523173] Lustre: 328575:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12734.709698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12740.666450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12740.677524] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12740.762921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12740.933791] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12750.638455] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12771.304988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12771.315711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12771.353041] Lustre: Skipped 1 previous similar message [12771.377138] Lustre: Skipped 1 previous similar message [12774.588489] Lustre: server umount lustre-OST0000 complete [12781.545902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12781.549994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12781.568299] Lustre: Skipped 1 previous similar message [12781.588126] Lustre: Skipped 1 previous similar message [12786.658779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12786.679611] Lustre: Skipped 1 previous similar message [12791.779339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12793.312300] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12793.628455] Lustre: server umount lustre-MDT0000 complete [12799.022327] LustreError: 328574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868976 with bad export cookie 12939991498152116576 [12799.027907] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12799.044133] LustreError: 328574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12799.511587] Lustre: server umount lustre-MDT0001 complete [12814.166816] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [12817.427916] Key type lgssc unregistered [12817.883727] LNet: 330489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12817.893150] LNetError: 330489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12817.904951] LNet: Removed LNI 192.168.202.135@tcp [12818.769105] Key type .llcrypt unregistered [12818.770698] Key type ._llcrypt unregistered [12844.836632] Key type ._llcrypt registered [12844.837777] Key type .llcrypt registered [12844.993365] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [12846.889518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12847.013570] alg: No test for adler32 (adler32-zlib) [12848.141920] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [12848.552285] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [12850.290686] Key type lgssc registered [12851.569842] Lustre: Echo OBD driver; http://www.lustre.org/ [12861.056932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12861.100833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12862.742954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12868.069114] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12872.175725] LustreError: 331764:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12872.196500] LustreError: 331764:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [12876.857103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12877.191708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12883.255136] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12893.493868] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12901.208503] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12910.546394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12911.046935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12917.844769] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12926.005109] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12947.265224] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12949.134797] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12954.100418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12954.108413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12954.143730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12954.152723] Lustre: Skipped 2 previous similar messages [12957.894465] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.35@tcp (stopping) [12959.204908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12959.209620] Lustre: Skipped 1 previous similar message [12960.016137] Lustre: server umount lustre-MDT0000 complete [12962.978091] LustreError: 334150:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12964.325124] LustreError: 331764:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12964.359717] LustreError: 331764:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [12964.580084] LustreError: 333608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869141 with bad export cookie 4619399566338702759 [12964.594025] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12964.601044] LustreError: 333608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12968.096660] LustreError: 331760:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12968.133100] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.35@tcp (stopping) [12968.139078] Lustre: Skipped 1 previous similar message [12969.446262] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12969.467831] Lustre: Skipped 2 previous similar messages [12971.347309] Lustre: server umount lustre-MDT0001 complete [12976.535804] Lustre: server umount lustre-OST0000 complete [12986.861554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12992.815225] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [12996.864441] Lustre: 335971:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.135@tcp [12999.657901] Lustre: server umount MGS complete [13010.612497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13011.422087] LustreError: 336543:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13011.465243] LustreError: 336543:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [13011.610485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13017.845248] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13021.164714] LustreError: 336544:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13021.185959] LustreError: 336544:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [13027.838424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13028.159972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13033.524470] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13044.568712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13044.771278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13051.229504] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13060.334336] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13062.316270] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13069.134078] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13071.271900] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13079.536638] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13079.550539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13079.565896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13079.570949] Lustre: Skipped 1 previous similar message [13081.569654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13081.585545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13083.740691] Lustre: server umount lustre-OST0000 complete [13089.765274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13089.785104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13089.808373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13089.821776] Lustre: Skipped 2 previous similar messages [13103.072155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13103.306775] Lustre: server umount lustre-MDT0000 complete [13106.149571] LustreError: 336543:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13106.177563] LustreError: 336543:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [13108.876232] LustreError: 338000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869285 with bad export cookie 4619399566338703599 [13108.878140] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13108.891185] LustreError: 338000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13109.518580] Lustre: server umount lustre-MDT0001 complete [13124.262841] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [13127.072469] Key type lgssc unregistered [13127.315245] LNet: 339984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13127.324700] LNetError: 339984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13128.361449] LNet: Removed LNI 192.168.202.135@tcp [13129.384834] Key type .llcrypt unregistered [13129.387923] Key type ._llcrypt unregistered [13151.770896] Key type ._llcrypt registered [13151.773412] Key type .llcrypt registered [13151.943533] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [13172.019189] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [13173.402971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13173.540695] alg: No test for adler32 (adler32-zlib) [13174.756472] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [13175.196098] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [13177.066792] Key type lgssc registered [13178.824698] Lustre: Echo OBD driver; http://www.lustre.org/ [13224.520666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13224.554820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13225.903530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13225.960654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13226.045544] Lustre: lustre-MDT0000: new disk, initializing [13226.124822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13226.153922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13231.521472] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13244.180767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13244.367496] Lustre: 343149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13244.448589] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13244.460342] Lustre: Skipped 1 previous similar message [13244.537586] Lustre: lustre-MDT0001: new disk, initializing [13244.672745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13244.727921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13244.739383] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13249.762229] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13261.604172] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13268.961311] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13277.252551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13277.518497] Lustre: lustre-OST0000: new disk, initializing [13277.522287] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13277.528705] Lustre: 345119:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13277.618256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13284.502163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13284.519958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13284.700243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13287.128100] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13298.168409] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13305.132707] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13305.491569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13311.430820] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13311.818483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13315.555063] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13315.573686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13315.588826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13315.596231] Lustre: Skipped 1 previous similar message [13320.678479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13321.114740] Lustre: server umount lustre-OST0000 complete [13325.795972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13325.805427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13325.809533] Lustre: Skipped 2 previous similar messages [13325.813860] Lustre: Skipped 1 previous similar message [13330.915434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13330.922308] Lustre: Skipped 2 previous similar messages [13331.900734] Lustre: server umount lustre-MDT0000 complete [13335.959732] LustreError: 343143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869512 with bad export cookie 15600928380657494662 [13335.965753] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13335.977242] LustreError: 343143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13336.272942] Lustre: server umount lustre-MDT0001 complete [13347.128368] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 18:25:21 (1787869521) [13366.581310] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [13376.024244] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [13419.993445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13420.234122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13420.260742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13420.362049] Lustre: lustre-MDT0000: new disk, initializing [13420.461215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13420.478817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13424.796537] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13435.964344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13436.109583] Lustre: 349884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13436.173019] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13436.178053] Lustre: Skipped 1 previous similar message [13436.283764] Lustre: lustre-MDT0001: new disk, initializing [13436.434135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13436.469494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13436.490531] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13441.559847] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13452.104408] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13458.417964] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13466.735626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13467.069830] Lustre: lustre-OST0000: new disk, initializing [13467.074417] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13467.080810] Lustre: 351854:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13467.187311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13474.119750] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [13476.456238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13476.473783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13476.576746] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13485.469361] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13522.922813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13522.925168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13522.930114] LustreError: Skipped 2 previous similar messages [13522.930483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13522.974254] Lustre: Skipped 1 previous similar message [13528.297367] Lustre: server umount lustre-OST0000 complete [13533.159455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13533.166575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13533.173682] Lustre: Skipped 1 previous similar message [13533.188082] Lustre: Skipped 4 previous similar messages [13539.527759] Lustre: server umount lustre-MDT0000 complete [13544.272606] LustreError: 352722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869721 with bad export cookie 15600928380657495544 [13544.279903] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13544.285236] LustreError: 352722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13544.473524] Lustre: server umount lustre-MDT0001 complete [13560.418969] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [13564.260370] Key type lgssc unregistered [13564.590642] LNet: 354030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13564.597444] LNetError: 354030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13564.625408] LNet: Removed LNI 192.168.202.135@tcp [13565.624545] Key type .llcrypt unregistered [13565.628795] Key type ._llcrypt unregistered [13667.285259] Key type ._llcrypt registered [13667.288814] Key type .llcrypt registered [13667.405738] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [13667.665272] Key type .llcrypt unregistered [13667.668598] Key type ._llcrypt unregistered [13748.623874] Key type ._llcrypt registered [13748.626839] Key type .llcrypt registered [13748.742502] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [13749.050516] Key type .llcrypt unregistered [13749.055496] Key type ._llcrypt unregistered [13783.267533] Key type ._llcrypt registered [13783.273209] Key type .llcrypt registered [13783.447991] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [13783.836815] Key type .llcrypt unregistered [13783.841768] Key type ._llcrypt unregistered [13814.294134] Key type ._llcrypt registered [13814.301912] Key type .llcrypt registered [13814.439452] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [13814.857523] Key type .llcrypt unregistered [13814.858963] Key type ._llcrypt unregistered [13902.969661] Key type ._llcrypt registered [13902.971695] Key type .llcrypt registered [13903.032332] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [13903.261281] Key type .llcrypt unregistered [13903.263099] Key type ._llcrypt unregistered [13937.267904] Key type ._llcrypt registered [13937.270820] Key type .llcrypt registered [13937.409737] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [13937.690341] Key type .llcrypt unregistered [13937.696388] Key type ._llcrypt unregistered [13972.994963] Key type ._llcrypt registered [13972.997461] Key type .llcrypt registered [13973.152361] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [13973.646530] Key type .llcrypt unregistered [13973.652600] Key type ._llcrypt unregistered [14009.175473] Key type ._llcrypt registered [14009.177831] Key type .llcrypt registered [14009.298187] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [14009.671420] Key type .llcrypt unregistered [14009.674649] Key type ._llcrypt unregistered [14043.358613] Key type ._llcrypt registered [14043.366146] Key type .llcrypt registered [14043.471349] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [14043.732507] Key type .llcrypt unregistered [14043.734274] Key type ._llcrypt unregistered [14074.628089] Key type ._llcrypt registered [14074.631154] Key type .llcrypt registered [14074.737785] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [14075.115435] Key type .llcrypt unregistered [14075.119471] Key type ._llcrypt unregistered [14142.630200] Key type ._llcrypt registered [14142.631724] Key type .llcrypt registered [14142.866839] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [14143.417554] Key type .llcrypt unregistered [14143.418983] Key type ._llcrypt unregistered [14175.474551] Key type ._llcrypt registered [14175.476057] Key type .llcrypt registered [14175.603370] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [14175.864315] Key type .llcrypt unregistered [14175.865709] Key type ._llcrypt unregistered [14209.741811] Key type ._llcrypt registered [14209.747717] Key type .llcrypt registered [14209.977184] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [14210.549682] Key type .llcrypt unregistered [14210.556153] Key type ._llcrypt unregistered [14269.109695] Key type ._llcrypt registered [14269.115548] Key type .llcrypt registered [14269.238543] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [14286.983689] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [14288.619106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14288.646800] alg: No test for adler32 (adler32-zlib) [14289.847869] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [14290.179813] LNet: Added LNI 192.168.202.135@tcp [8/256/0/180] [14291.896217] Key type lgssc registered [14293.304639] Lustre: Echo OBD driver; http://www.lustre.org/ [14330.388206] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14330.413815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14331.694619] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14331.717668] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14331.818642] Lustre: tst250fs-MDT0000: new disk, initializing [14331.982122] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14332.011571] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14336.766725] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [14348.956636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14349.158488] Lustre: 374451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14349.238927] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14349.243435] Lustre: Skipped 1 previous similar message [14349.370426] Lustre: tst250fs-MDT0001: new disk, initializing [14349.447852] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14349.483294] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14349.500153] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14355.369272] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [14361.427909] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14369.030935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14369.211558] Lustre: tst250fs-OST0000: new disk, initializing [14369.214393] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14369.218504] Lustre: 376084:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14369.303931] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14374.978937] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14374.984411] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14375.140546] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14375.391145] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [14387.286532] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14387.634146] Lustre: tst250fs-OST0001: new disk, initializing [14387.645316] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14387.657458] Lustre: 376956:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14387.835875] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14396.112383] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all [14397.233528] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14397.267627] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14397.593040] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14405.176263] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14408.629325] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14408.635569] Lustre: Skipped 1 previous similar message [14431.713514] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14431.719710] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14431.730564] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14433.253894] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14433.256648] Lustre: tst250fs-MDT0000-lwp-OST0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14433.271761] Lustre: Skipped 1 previous similar message [14433.298441] Lustre: Skipped 1 previous similar message [14437.920849] Lustre: server umount tst250fs-MDT0000 complete [14441.707360] LustreError: 375246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870618 with bad export cookie 13202271541158361092 [14441.722224] LustreError: 375246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14441.726824] LustreError: MGC192.168.202.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14442.094767] Lustre: server umount tst250fs-MDT0001 complete [14456.614466] Lustre: server umount tst250fs-OST0000 complete [14459.361107] Lustre: 372675:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787870620/real 1787870620] req@ffff89c50f276940 x1874718062877568/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787870636 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14459.383588] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14459.394515] Lustre: Skipped 1 previous similar message [14460.768938] Lustre: server umount tst250fs-OST0001 complete [14484.343388] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [14493.243435] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [14529.533776] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 18:45:04 (1787870704) [14531.523840] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14551.701251] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_hostid [14564.591664] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing load_modules_local [14601.904911] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14334 sec ========= 18:46:16 (1787870776) [14604.106590] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:46:18 (1787870778) === [14609.642718] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:46:23 (1787870783) === [14651.287517] Lustre: DEBUG MARKER: oleg235-server.virtnet: executing unload_modules_local [14653.749785] Key type lgssc unregistered [14654.036579] LNet: 386090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14654.044056] LNetError: 386090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14655.081910] LNet: Removed LNI 192.168.202.135@tcp [14655.842167] Key type .llcrypt unregistered [14655.845569] Key type ._llcrypt unregistered