[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 476624452 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003271] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008410] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.010017] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.011012] pid_max: default: 32768 minimum: 301 [ 0.012102] LSM: Security Framework initializing [ 0.013049] Yama: becoming mindful. [ 0.014047] SELinux: Initializing. [ 0.015064] *** VALIDATE selinux *** [ 0.022719] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026631] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030090] *** VALIDATE tmpfs *** [ 0.031481] *** VALIDATE proc *** [ 0.032261] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.035220] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038028] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042310] debug: unmapping init [mem 0xffffffffba259000-0xffffffffba260fff] [ 0.044948] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045657] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046026] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048015] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050013] ... max period: 00007fffffffffff [ 0.051013] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053310] rcu: Hierarchical SRCU implementation. [ 0.055438] smp: Bringing up secondary CPUs ... [ 0.056528] x86: Booting SMP configuration: [ 0.057027] .... node #0, CPUs: #1 #2 #3 [ 0.060482] smp: Brought up 1 node, 4 CPUs [ 0.062013] smpboot: Max logical packages: 1 [ 0.063020] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.139021] node 0 deferred pages initialised in 74ms [ 0.141094] devtmpfs: initialized [ 0.143262] x86/mm: Memory block size: 128MB [ 0.147037] gcov: version magic: 0x41383552 [ 0.149161] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153099] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155270] pinctrl core: initialized pinctrl subsystem [ 0.157159] [ 0.157641] ************************************************************* [ 0.159013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161011] ** ** [ 0.163013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165012] ** ** [ 0.167011] ** This means that this kernel is built to expose internal ** [ 0.168010] ** IOMMU data structures, which may compromise security on ** [ 0.170011] ** your system. ** [ 0.172017] ** ** [ 0.173010] ** If you see this message and you are not debugging the ** [ 0.175012] ** kernel, report this immediately to your vendor! ** [ 0.177011] ** ** [ 0.179013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.181013] ************************************************************* [ 0.183454] NET: Registered protocol family 16 [ 0.185486] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.187057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.189052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.193041] cpuidle: using governor menu [ 0.194790] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.197478] PCI: Using configuration type 1 for base access [ 0.199125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.206094] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.207021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.209108] cryptd: max_cpu_qlen set to 1000 [ 0.212236] ACPI: Added _OSI(Module Device) [ 0.214014] ACPI: Added _OSI(Processor Device) [ 0.215012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.217013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.221945] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.228486] ACPI: Interpreter enabled [ 0.230068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.232012] ACPI: Using IOAPIC for interrupt routing [ 0.234107] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.237418] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.248887] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.250060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.253023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.256076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.260417] acpiphp: Slot [2] registered [ 0.262144] acpiphp: Slot [5] registered [ 0.263133] acpiphp: Slot [6] registered [ 0.265168] acpiphp: Slot [7] registered [ 0.266114] acpiphp: Slot [8] registered [ 0.268082] acpiphp: Slot [9] registered [ 0.269096] acpiphp: Slot [10] registered [ 0.270120] acpiphp: Slot [3] registered [ 0.272100] acpiphp: Slot [4] registered [ 0.274122] acpiphp: Slot [11] registered [ 0.275094] acpiphp: Slot [12] registered [ 0.277110] acpiphp: Slot [13] registered [ 0.278101] acpiphp: Slot [14] registered [ 0.280101] acpiphp: Slot [15] registered [ 0.281105] acpiphp: Slot [16] registered [ 0.283135] acpiphp: Slot [17] registered [ 0.285102] acpiphp: Slot [18] registered [ 0.286148] acpiphp: Slot [19] registered [ 0.288196] acpiphp: Slot [20] registered [ 0.289142] acpiphp: Slot [21] registered [ 0.291193] acpiphp: Slot [22] registered [ 0.292190] acpiphp: Slot [23] registered [ 0.294165] acpiphp: Slot [24] registered [ 0.296152] acpiphp: Slot [25] registered [ 0.298121] acpiphp: Slot [26] registered [ 0.299119] acpiphp: Slot [27] registered [ 0.301102] acpiphp: Slot [28] registered [ 0.302089] acpiphp: Slot [29] registered [ 0.304156] acpiphp: Slot [30] registered [ 0.305090] acpiphp: Slot [31] registered [ 0.306099] PCI host bridge to bus 0000:00 [ 0.308025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.309000] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.310025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.312024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.315022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.318027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.319178] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.322900] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.326015] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.339746] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.344958] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.348036] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.350028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.354032] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.356690] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.360101] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.363064] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.366864] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.372020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.384000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.389018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.395030] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.403023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.411026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.430032] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.445719] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.451017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.458017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.474029] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.483201] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.493025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.503026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.525028] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.535770] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.541013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.548018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.571026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.580000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.585019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.590018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.605016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.615252] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.624176] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.631026] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.652025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.663295] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.665400] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.668306] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.670375] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.673232] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.678060] iommu: Default domain type: Passthrough [ 0.680642] SCSI subsystem initialized [ 0.682204] ACPI: bus type USB registered [ 0.684123] usbcore: registered new interface driver usbfs [ 0.686122] usbcore: registered new interface driver hub [ 0.687144] usbcore: registered new device driver usb [ 0.689227] pps_core: LinuxPPS API ver. 1 registered [ 0.691015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.694073] PTP clock support registered [ 0.696258] EDAC MC: Ver: 3.0.0 [ 0.698320] PCI: Using ACPI for IRQ routing [ 0.700915] NetLabel: Initializing [ 0.702018] NetLabel: domain hash size = 128 [ 0.704013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.705077] NetLabel: unlabeled traffic allowed by default [ 0.708110] vgaarb: loaded [ 0.709285] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.711018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.715420] clocksource: Switched to clocksource kvm-clock [ 0.834958] VFS: Disk quotas dquot_6.6.0 [ 0.836372] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.838661] *** VALIDATE ramfs *** [ 0.839756] *** VALIDATE hugetlbfs *** [ 0.841250] pnp: PnP ACPI init [ 0.843420] pnp: PnP ACPI: found 6 devices [ 0.863518] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.866610] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.868882] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.870694] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.872373] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.874356] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.877257] NET: Registered protocol family 2 [ 0.879762] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.884758] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.888134] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.894175] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.897888] TCP: Hash tables configured (established 65536 bind 65536) [ 0.901387] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.905086] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.908252] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.911640] NET: Registered protocol family 1 [ 0.913763] RPC: Registered named UNIX socket transport module. [ 0.916197] RPC: Registered udp transport module. [ 0.918119] RPC: Registered tcp transport module. [ 0.920127] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.922729] NET: Registered protocol family 44 [ 0.924504] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.926173] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.927553] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.929730] PCI: CLS 0 bytes, default 64 [ 0.932129] Unpacking initramfs... [ 2.372669] debug: unmapping init [mem 0xffff9a78fcc54000-0xffff9a78fffbffff] [ 2.376737] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.379387] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.382557] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.875323] Initialise system trusted keyrings [ 2.876725] Key type blacklist registered [ 2.878497] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.887508] zbud: loaded [ 2.890542] *** VALIDATE nfs *** [ 2.891651] *** VALIDATE nfs4 *** [ 2.893132] pstore: using deflate compression [ 2.896442] Platform Keyring initialized [ 3.015826] NET: Registered protocol family 38 [ 3.017785] Key type asymmetric registered [ 3.019921] Asymmetric key parser 'x509' registered [ 3.022218] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.025815] io scheduler mq-deadline registered [ 3.027337] io scheduler kyber registered [ 3.029269] io scheduler bfq registered [ 3.031495] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.035079] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.038264] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.047953] ACPI: Power Button [PWRF] [ 3.053487] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.059787] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.071904] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.078515] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.091248] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.120759] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.149631] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.157851] Non-volatile memory driver v1.3 [ 3.159666] Linux agpgart interface v0.103 [ 3.196283] virtio_blk virtio1: [vda] 145224 512-byte logical blocks (74.4 MB/70.9 MiB) [ 3.199125] vda: detected capacity change from 0 to 74354688 [ 3.212274] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.215170] vdb: detected capacity change from 0 to 1073741824 [ 3.228714] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.231716] vdc: detected capacity change from 0 to 2621440000 [ 3.244254] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.247310] vdd: detected capacity change from 0 to 2621440000 [ 3.260883] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.263830] vde: detected capacity change from 0 to 4294967296 [ 3.278110] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.281664] vdf: detected capacity change from 0 to 4294967296 [ 3.290651] libphy: Fixed MDIO Bus: probed [ 3.306055] usbcore: registered new interface driver usbserial_generic [ 3.307707] usbserial: USB Serial support registered for generic [ 3.309377] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.313340] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.314946] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.317453] mousedev: PS/2 mouse device common for all mice [ 3.320310] rtc_cmos 00:05: RTC can wake from S4 [ 3.323337] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.327099] rtc_cmos 00:05: registered as rtc0 [ 3.329951] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.330136] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.332850] intel_pstate: CPU model not supported [ 3.338813] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.343164] hid: raw HID events driver (C) Jiri Kosina [ 3.345764] usbcore: registered new interface driver usbhid [ 3.348080] usbhid: USB HID core driver [ 3.349762] drop_monitor: Initializing network drop monitor service [ 3.352410] Initializing XFRM netlink socket [ 3.354778] NET: Registered protocol family 10 [ 3.357813] Segment Routing with IPv6 [ 3.359554] NET: Registered protocol family 17 [ 3.361920] mpls_gso: MPLS GSO support [ 3.367778] RAS: Correctable Errors collector initialized. [ 3.370224] AVX version of gcm_enc/dec engaged. [ 3.372124] AES CTR mode by8 optimization enabled [ 3.451202] sched_clock: Marking stable (3451177045, 0)->(4385349792, -934172747) [ 3.455599] registered taskstats version 1 [ 3.458497] Loading compiled-in X.509 certificates [ 3.460584] zswap: loaded using pool lzo/zbud [ 3.484544] Key type big_key registered [ 3.495582] Key type encrypted registered [ 3.497150] ima: No TPM chip found, activating TPM-bypass! [ 3.499485] ima: Allocated hash algorithm: sha1 [ 3.501365] ima: No architecture policies found [ 3.503569] evm: Initialising EVM extended attributes: [ 3.505992] evm: security.selinux [ 3.507364] evm: security.ima [ 3.508611] evm: security.capability [ 3.510254] evm: HMAC attrs: 0x1 [ 3.513110] rtc_cmos 00:05: setting system clock to 2026-06-23 19:11:37 UTC (1782241897) [ 3.520050] debug: unmapping init [mem 0xffffffffbb203000-0xffffffffbb3fffff] [ 3.523250] debug: unmapping init [mem 0xffffffffb9f82000-0xffffffffba258fff] [ 3.536235] Write protecting the kernel read-only data: 28672k [ 3.540028] debug: unmapping init [mem 0xffffffffb8603000-0xffffffffb87fffff] [ 3.542317] debug: unmapping init [mem 0xffffffffb8f14000-0xffffffffb8ffffff] [ 3.581201] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.589047] systemd[1]: Detected virtualization kvm. [ 3.590640] systemd[1]: Detected architecture x86-64. [ 3.592197] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.617178] systemd[1]: No hostname configured. [ 3.619100] systemd[1]: Set hostname to . [ 3.621056] random: systemd: uninitialized urandom read (16 bytes read) [ 3.623162] systemd[1]: Initializing machine ID from random generator. [ 3.770643] random: systemd: uninitialized urandom read (16 bytes read) [ 3.772935] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.776751] random: systemd: uninitialized urandom read (16 bytes read) [ 3.779545] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.784147] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.469186] device-mapper: uevent: version 1.0.3 [ 4.470624] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.269970] random: fast init done [ 5.283578] virtio_net virtio0 ens2: renamed from eth0 [ 5.328349] scsi host0: ata_piix [ 5.337505] scsi host1: ata_piix [ 5.339140] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.341186] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.148475] dracut-initqueue[575]: RTNETLINK answers: File exists [ 10.015651] random: crng init done [ 10.017091] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.516363] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.714977] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.993733] SELinux: Disabled at runtime. [ 12.058394] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.066315] systemd[1]: Detected virtualization kvm. [ 12.068679] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.676480] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.681039] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.685575] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.690098] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.698438] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.705692] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.713203] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 12.738737] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.193943] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.596625] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.601721] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.785864] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.797919] EDAC sbridge: Ver: 1.1.2 [ 15.526626] Key type dns_resolver registered [ 15.835562] NFS: Registering the id_resolver key type [ 15.837624] Key type id_resolver registered [ 15.839254] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg435-server login: [ 42.164139] libcfs: loading out-of-tree module taints kernel. [ 42.186227] Key type ._llcrypt registered [ 42.187823] Key type .llcrypt registered [ 42.244892] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 51.631753] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 52.379624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.389987] alg: No test for adler32 (adler32-zlib) [ 53.451134] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 53.821697] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 55.480231] Key type lgssc registered [ 56.873611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 72.588736] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 91.727133] hrtimer: interrupt took 1465707 ns [ 112.838633] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 127.634743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 127.674380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 128.975857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 129.031571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 129.152535] Lustre: lustre-MDT0000: new disk, initializing [ 129.280658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 129.296964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 134.224638] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 148.448892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 148.585513] Lustre: 6482:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 148.644845] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 148.651880] Lustre: Skipped 1 previous similar message [ 148.742494] Lustre: lustre-MDT0001: new disk, initializing [ 148.822853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 148.871899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 148.883411] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 153.879749] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 159.022489] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 171.360741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 171.702921] Lustre: lustre-OST0000: new disk, initializing [ 171.712505] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 171.725762] Lustre: 8418:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 171.893303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 179.348094] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.776924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 179.788318] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 179.926334] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 197.101827] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.325790] Lustre: lustre-OST0001: new disk, initializing [ 197.331356] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 197.359465] Lustre: 9489:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 197.502080] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 205.633836] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.900973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 205.920949] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 206.036844] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 216.210642] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 223.793313] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 229.540163] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing check_logdir /tmp/testlogs/ [ 236.243315] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing yml_node [ 241.203504] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 244.202174] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 246.673996] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 248.392641] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 23 15:15:41 EDT 2026 [ 269.864806] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 271.808187] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 288.225891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.227839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 288.249450] Lustre: Skipped 1 previous similar message [ 293.344864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 293.358092] Lustre: Skipped 6 previous similar messages [ 294.055406] Lustre: server umount lustre-MDT0000 complete [ 297.889051] LustreError: 6474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782242191 with bad export cookie 17403946881802604047 [ 297.889773] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 297.904209] LustreError: 6474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 298.465577] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.476118] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 298.482750] Lustre: Skipped 3 previous similar messages [ 298.493067] Lustre: Skipped 1 previous similar message [ 302.501259] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 302.508939] Lustre: Skipped 1 previous similar message [ 307.617244] LustreError: 9737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.617869] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 307.641309] LustreError: 9737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 307.668296] Lustre: Skipped 1 previous similar message [ 312.288173] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 312.568378] Lustre: server umount lustre-MDT0001 complete [ 316.324377] Lustre: server umount lustre-OST0000 complete [ 319.774640] Lustre: server umount lustre-OST0001 complete [ 325.865206] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 334.926988] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 370.832566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 371.085320] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 371.118698] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 371.212647] Lustre: lustre-MDT0000: new disk, initializing [ 371.300222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 371.322599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 375.360120] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 385.775380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 385.865642] Lustre: 15046:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 385.929062] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 385.932788] Lustre: Skipped 1 previous similar message [ 386.014801] Lustre: lustre-MDT0001: new disk, initializing [ 386.108587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 386.150513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 386.165469] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 390.438284] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 399.625666] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 404.536661] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 410.629583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 410.853973] Lustre: lustre-OST0000: new disk, initializing [ 410.860506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 410.871413] Lustre: 17012:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 410.983939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 412.065082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 412.090721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 412.227192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 416.709134] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 425.474158] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 429.492956] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 429.732313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 433.775891] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 434.022699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 437.731643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 437.734264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 437.736810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 437.757474] Lustre: Skipped 1 previous similar message [ 450.016172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 450.188218] Lustre: server umount lustre-OST0000 complete [ 457.700582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 457.711085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 457.722844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 457.728989] Lustre: Skipped 5 previous similar messages [ 459.692221] Lustre: server umount lustre-MDT0000 complete [ 463.343547] LustreError: 15054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 463.349500] LustreError: 15038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782242357 with bad export cookie 17403946881802605580 [ 463.354733] LustreError: 15054:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 463.359123] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 463.401481] LustreError: 15038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 463.736237] Lustre: server umount lustre-MDT0001 complete [ 469.573937] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 15:19:22 (1782242362) [ 489.193722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.646042] LustreError: 19888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 489.744365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 494.126960] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 495.073242] LustreError: 19889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.173804] LustreError: 19888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 502.890736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 507.503911] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 514.636453] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 520.429993] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 522.685089] Lustre: Failing over lustre-MDT0000 [ 522.893626] Lustre: server umount lustre-MDT0000 complete [ 523.746667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 523.767164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 523.786655] Lustre: Skipped 1 previous similar message [ 523.797866] LustreError: 19889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.129256] Lustre: 3614:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782242418/real 1782242418] req@ffff9a79514d3800 x1868816131977856/t0(0) o400->MGC192.168.204.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782242434 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 540.168607] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 540.185535] LustreError: 20616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.205983] LustreError: 20616:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 543.185269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 550.371818] LustreError: 3612:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9a7974c11180 x1868816131980672/t0(0) o250->MGC192.168.204.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 555.507323] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 556.010047] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 556.024990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 556.058684] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 565.905779] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 570.338502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.360575] Lustre: Skipped 1 previous similar message [ 570.369090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 570.378132] Lustre: Skipped 3 previous similar messages [ 576.244671] Lustre: server umount lustre-MDT0000 complete [ 578.531196] LustreError: 19885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.548977] LustreError: 19885:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 579.687262] LustreError: 19870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782242473 with bad export cookie 17403946881802606994 [ 579.693989] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 579.701635] LustreError: 19870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 579.963402] Lustre: server umount lustre-MDT0001 complete [ 588.077519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 588.703251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 588.710947] Lustre: Skipped 2 previous similar messages [ 592.660360] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 600.611269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 605.307343] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 612.183813] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 617.582732] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 619.880094] Lustre: Failing over lustre-MDT0000 [ 620.285719] Lustre: server umount lustre-MDT0000 complete [ 621.539432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 621.547490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 621.563135] Lustre: Skipped 1 previous similar message [ 638.767771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 638.840272] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.111665] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 643.373674] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 644.066287] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 644.070313] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 644.084641] Lustre: Skipped 1 previous similar message [ 644.109795] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 653.505452] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 658.403750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 658.423585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 658.424529] Lustre: Skipped 2 previous similar messages [ 658.435165] Lustre: Skipped 3 previous similar messages [ 663.799762] Lustre: server umount lustre-MDT0000 complete [ 668.196522] LustreError: 23210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782242562 with bad export cookie 17403946881802607932 [ 668.210739] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 668.544518] Lustre: server umount lustre-MDT0001 complete [ 678.487358] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 15:22:51 (1782242571) [ 695.611806] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 705.859734] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 742.890378] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 752.962568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 753.132742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 753.172984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 753.248822] Lustre: lustre-MDT0000: new disk, initializing [ 753.344029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 753.352871] Lustre: Skipped 2 previous similar messages [ 753.392869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 757.943428] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 768.092443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 768.197219] Lustre: 29601:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 768.497201] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 772.240565] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 780.280968] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 785.982982] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 792.050940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 792.284236] Lustre: 31566:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 794.246616] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 794.303465] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 794.341825] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 794.368707] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 798.359432] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 808.597867] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 818.638916] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 840.673924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 840.682761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 840.687047] LustreError: Skipped 1 previous similar message [ 840.687743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 840.687747] Lustre: Skipped 2 previous similar messages [ 840.719090] Lustre: Skipped 1 previous similar message [ 843.118786] Lustre: server umount lustre-OST0000 complete [ 850.401877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 853.356975] Lustre: server umount lustre-MDT0000 complete [ 856.034165] LustreError: 29610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 856.059689] LustreError: 29610:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 857.523060] LustreError: 29593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782242751 with bad export cookie 17403946881802608338 [ 857.535467] LustreError: 29593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 857.535569] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 858.095450] Lustre: server umount lustre-MDT0001 complete [ 870.556873] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 874.210901] Key type lgssc unregistered [ 874.654542] LNet: 33545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.661964] LNetError: 33545:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.677402] LNet: Removed LNI 192.168.204.135@tcp [ 875.681495] Key type .llcrypt unregistered [ 875.683933] Key type ._llcrypt unregistered [ 899.291371] Key type ._llcrypt registered [ 899.293661] Key type .llcrypt registered [ 899.382464] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 900.265636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 900.427065] alg: No test for adler32 (adler32-zlib) [ 901.541741] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 901.884768] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 903.640280] Key type lgssc registered [ 904.855539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 913.435048] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 913.459553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 915.024763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 919.297367] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 924.132955] LustreError: 34961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 924.158374] LustreError: 34961:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 928.231803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 928.671688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 933.305790] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 941.022359] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 947.588235] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 954.553045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 954.790488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 961.411561] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 969.305503] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 973.237539] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 989.489740] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 15:28:02 (1782242882) [ 991.872517] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 993.861926] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1000.990416] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 15:28:14 (1782242894) [ 1009.649448] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 15:28:23 (1782242903) [ 1010.704541] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1020.005857] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1020.014921] Lustre: Skipped 2 previous similar messages [ 1021.739524] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 15:28:35 (1782242915) [ 1025.196186] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1025.207740] Lustre: Skipped 3 previous similar messages [ 1045.581192] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1045.592964] Lustre: Skipped 6 previous similar messages [ 1049.300383] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 15:29:01 (1782242941) [ 1053.111908] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1057.022658] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1066.588402] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 15:29:19 (1782242959) [ 1069.053621] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1069.057994] Lustre: Skipped 3 previous similar messages [ 1084.008243] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 15:29:36 (1782242976) [ 1101.287791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1101.298557] Lustre: Skipped 1 previous similar message [ 1101.302860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1101.303147] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1101.309705] Lustre: Skipped 1 previous similar message [ 1106.403876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1106.410288] Lustre: Skipped 1 previous similar message [ 1111.546713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1111.556097] Lustre: Skipped 1 previous similar message [ 1113.056287] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1113.236919] Lustre: server umount lustre-OST0000 complete [ 1118.177205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1118.193784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1118.223186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1121.772383] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1123.615734] Lustre: server umount lustre-MDT0000 complete [ 1126.881505] LustreError: 34961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1126.915146] LustreError: 34961:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1127.811349] LustreError: 34940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782243021 with bad export cookie 3533463946465403731 [ 1127.817277] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1127.821453] LustreError: 34940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1128.374252] Lustre: server umount lustre-MDT0001 complete [ 1139.720884] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 1142.159873] Key type lgssc unregistered [ 1142.430644] LNet: 42819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1142.450294] LNetError: 42819:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1142.480733] LNet: Removed LNI 192.168.204.135@tcp [ 1143.379910] Key type .llcrypt unregistered [ 1143.387904] Key type ._llcrypt unregistered [ 1162.239345] Key type ._llcrypt registered [ 1162.244567] Key type .llcrypt registered [ 1162.383789] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 1178.550382] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 1179.822714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1179.866089] alg: No test for adler32 (adler32-zlib) [ 1181.003145] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 1181.319620] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 1182.992184] Key type lgssc registered [ 1184.220357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1221.327962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1221.399175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1222.662073] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1222.699157] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1222.799783] Lustre: lustre-MDT0000: new disk, initializing [ 1222.917618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1222.945754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1227.837696] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1239.905319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1240.126294] Lustre: 45977:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1240.237944] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1240.258619] Lustre: Skipped 1 previous similar message [ 1240.372304] Lustre: lustre-MDT0001: new disk, initializing [ 1240.519965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1240.555393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1240.567460] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1245.400764] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1256.094736] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1262.935369] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1271.107361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1271.426896] Lustre: lustre-OST0000: new disk, initializing [ 1271.433607] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1271.445825] Lustre: 47947:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1271.570569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1279.120820] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1281.129549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1281.140626] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1281.288208] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1290.704473] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1296.338556] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1296.543750] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1301.571143] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1301.817434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1307.105617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1307.112675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1307.128081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1307.150155] Lustre: Skipped 1 previous similar message [ 1312.235768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1312.243100] Lustre: Skipped 2 previous similar messages [ 1317.349791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1317.368522] Lustre: Skipped 1 previous similar message [ 1318.370576] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1318.524633] Lustre: server umount lustre-OST0000 complete [ 1322.472604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1322.485450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1322.511643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1322.519041] Lustre: Skipped 1 previous similar message [ 1327.590845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1327.599805] Lustre: Skipped 1 previous similar message [ 1328.975055] Lustre: server umount lustre-MDT0000 complete [ 1333.212707] LustreError: 45969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782243227 with bad export cookie 14544626848896446964 [ 1333.224182] LustreError: 45969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1333.226264] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1333.828153] Lustre: server umount lustre-MDT0001 complete [ 1345.142605] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 15:33:57 (1782243237) [ 1353.413851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1353.959470] LustreError: 50410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1353.992434] LustreError: 50410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1354.063515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1358.676877] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1359.333582] LustreError: 50411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1359.357735] LustreError: 50411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1363.437747] LustreError: 50410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1367.928645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1368.463460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1372.959764] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1381.407742] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1388.402187] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1395.773368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1396.249267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1403.023587] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1411.140981] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1419.077359] LustreError: 52939:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1420.069840] Lustre: Modifying parameter general.timeout=1 in log params [ 1421.951642] Lustre: Modifying parameter general.timeout=3 in log params [ 1421.958223] Lustre: Skipped 1 previous similar message [ 1424.704235] Lustre: Modifying parameter general.timeout=6 in log params [ 1424.707677] Lustre: Skipped 2 previous similar messages [ 1428.739834] Lustre: Modifying parameter general.timeout=11 in log params [ 1428.745285] Lustre: Skipped 4 previous similar messages [ 1437.566926] Lustre: Modifying parameter general.timeout=22 in log params [ 1437.578122] Lustre: Skipped 10 previous similar messages [ 1453.657208] Lustre: Modifying parameter general.timeout=39 in log params [ 1453.665407] Lustre: Skipped 16 previous similar messages [ 1485.684928] Lustre: Modifying parameter general.timeout=73 in log params [ 1485.688706] Lustre: Skipped 33 previous similar messages [ 1544.922702] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 15:37:18 (1782243438) [ 1547.559244] Lustre: 59881:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1549.400169] Lustre: 59979:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1550.546431] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 1550.576251] Lustre: Skipped 60 previous similar messages [ 1551.515795] Lustre: 60077:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1557.160994] Lustre: 60392:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1557.184215] Lustre: 60392:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 2 previous similar messages [ 1590.044846] Lustre: 61758:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1590.081846] LustreError: 61758:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1595.902722] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 15:38:08 (1782243488) [ 1610.224921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1610.227954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1610.243827] Lustre: Skipped 3 previous similar messages [ 1610.250638] Lustre: Skipped 1 previous similar message [ 1615.660619] Lustre: server umount lustre-MDT0000 complete [ 1619.900506] LustreError: 54770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782243513 with bad export cookie 14544626848896447846 [ 1619.903109] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1619.915951] LustreError: 54770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1620.341580] Lustre: server umount lustre-MDT0001 complete [ 1635.848319] Lustre: server umount lustre-OST0000 complete [ 1662.341443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.577326] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1662.845046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1667.246612] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1675.981984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1676.068927] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1676.107338] Lustre: 64293:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1681.097642] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1688.693962] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1693.777754] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1700.088816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1700.277689] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1700.566250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1700.580469] Lustre: Skipped 1 previous similar message [ 1706.905374] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1714.295180] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1715.939568] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1715.947554] Lustre: Skipped 19 previous similar messages [ 1716.043916] Lustre: 66860:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1725.409240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1725.419253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1725.436088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1725.439979] Lustre: Skipped 3 previous similar messages [ 1729.353851] Lustre: server umount lustre-OST0000 complete [ 1736.163144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1736.183551] Lustre: Skipped 2 previous similar messages [ 1739.992284] Lustre: server umount lustre-MDT0000 complete [ 1741.281816] LustreError: 64299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1741.313737] LustreError: 64299:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1743.869904] LustreError: 64286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782243637 with bad export cookie 14544626848896450954 [ 1743.874882] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1743.887962] LustreError: 64286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1744.448385] Lustre: server umount lustre-MDT0001 complete [ 1755.776507] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 1758.418116] Key type lgssc unregistered [ 1758.687741] LNet: 68071:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1758.697982] LNetError: 68071:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1759.724553] LNet: Removed LNI 192.168.204.135@tcp [ 1760.623673] Key type .llcrypt unregistered [ 1760.627053] Key type ._llcrypt unregistered [ 1789.401994] Key type ._llcrypt registered [ 1789.412477] Key type .llcrypt registered [ 1789.617362] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 1790.681757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1790.732793] alg: No test for adler32 (adler32-zlib) [ 1791.805729] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 1792.029356] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 1793.680234] Key type lgssc registered [ 1794.912792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1803.277607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1803.320137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1804.743856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1809.471743] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1815.012333] LustreError: 69486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1815.036037] LustreError: 69486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1818.432122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1818.882493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1823.247237] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1831.065899] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1836.983905] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1845.328730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1846.143679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1853.706643] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1861.872296] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1872.632833] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1884.131189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1884.141772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1884.155683] Lustre: Skipped 2 previous similar messages [ 1884.165468] Lustre: Skipped 2 previous similar messages [ 1888.943341] Lustre: server umount lustre-MDT0000 complete [ 1889.253306] LustreError: 69487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1889.275411] LustreError: 69487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1892.964993] LustreError: 72170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782243786 with bad export cookie 2406539028594695111 [ 1892.970514] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1892.973865] LustreError: 72170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1893.415768] Lustre: server umount lustre-MDT0001 complete [ 1907.860453] Lustre: server umount lustre-OST0000 complete [ 1934.458486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1934.830772] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1934.861769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1935.191927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1940.187838] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1951.313769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1951.486912] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1951.589920] Lustre: 74358:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1952.366596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1957.312314] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1965.085607] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1970.573643] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1977.678614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1977.925464] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1978.406125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1984.281186] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1992.081652] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1993.977926] Lustre: 76935:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1995.900588] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2005.473721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2005.482574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.502438] Lustre: Skipped 1 previous similar message [ 2005.502714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2008.123995] Lustre: server umount lustre-OST0000 complete [ 2013.672605] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2013.677900] LustreError: Skipped 1 previous similar message [ 2013.681470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2013.687980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2013.690797] Lustre: Skipped 1 previous similar message [ 2015.714574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2015.735810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2015.748810] Lustre: Skipped 1 previous similar message [ 2018.560750] Lustre: server umount lustre-MDT0000 complete [ 2020.834625] LustreError: 75446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2020.874802] LustreError: 75446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2025.021088] LustreError: 74352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782243919 with bad export cookie 2406539028594695958 [ 2025.026915] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2025.057913] LustreError: 74352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2026.533303] Lustre: server umount lustre-MDT0001 complete [ 2042.329459] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2045.131233] Key type lgssc unregistered [ 2045.392586] LNet: 78147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2045.400206] LNetError: 78147:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2045.435727] LNet: Removed LNI 192.168.204.135@tcp [ 2046.111845] Key type .llcrypt unregistered [ 2046.114499] Key type ._llcrypt unregistered [ 2075.786564] Key type ._llcrypt registered [ 2075.789977] Key type .llcrypt registered [ 2075.945993] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2077.285184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2077.425406] alg: No test for adler32 (adler32-zlib) [ 2078.719297] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 2079.038944] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2080.728532] Key type lgssc registered [ 2082.121762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2091.475805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2091.500498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2093.185410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2097.700052] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2103.792882] LustreError: 79562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2103.810493] LustreError: 79562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2107.635140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2108.429170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2113.678487] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2123.320284] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2130.564513] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2140.524906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2141.502363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2150.060823] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2160.577756] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2171.138834] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2172.137096] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2173.238914] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2175.401901] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2175.407028] Lustre: Skipped 1 previous similar message [ 2179.480300] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2179.485244] Lustre: Skipped 3 previous similar messages [ 2188.446929] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2188.454865] Lustre: Skipped 7 previous similar messages [ 2205.739555] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2205.750423] Lustre: Skipped 14 previous similar messages [ 2238.232111] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2238.237639] Lustre: Skipped 28 previous similar messages [ 2292.546287] Lustre: 87679:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2295.393696] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 15:49:48 (1782244188) [ 2297.467365] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2299.285894] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 15:49:52 (1782244192) [ 2307.554256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2307.560359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2307.561493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2307.582282] Lustre: Skipped 1 previous similar message [ 2309.802595] Lustre: server umount lustre-OST0000 complete [ 2317.794502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2317.812688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2317.820021] Lustre: Skipped 1 previous similar message [ 2322.380929] Lustre: server umount lustre-MDT0000 complete [ 2322.914475] LustreError: 79559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2322.952384] LustreError: 79559:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2327.019112] LustreError: 83244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782244221 with bad export cookie 1597145523800376564 [ 2327.031665] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2327.037027] LustreError: 83244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2327.472221] Lustre: server umount lustre-MDT0001 complete [ 2341.176455] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2344.308533] Key type lgssc unregistered [ 2344.757414] LNet: 88928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2344.764732] LNetError: 88928:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2344.779591] LNet: Removed LNI 192.168.204.135@tcp [ 2345.819214] Key type .llcrypt unregistered [ 2345.823339] Key type ._llcrypt unregistered [ 2351.228722] Key type ._llcrypt registered [ 2351.231812] Key type .llcrypt registered [ 2351.427118] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2357.140835] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules [ 2358.511485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2358.528422] alg: No test for adler32 (adler32-zlib) [ 2359.843857] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 2360.374647] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2362.250546] Key type lgssc registered [ 2364.175281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2365.712776] LustreError: 90178:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2384.858968] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2388.422415] LustreError: 90178:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2396.032702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2396.064628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2397.876048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2403.326455] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2407.400183] LustreError: 91175:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2407.423983] LustreError: 91175:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2412.523404] LustreError: 91174:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2414.084803] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 15:51:46 (1782244306) [ 2417.637822] LustreError: 91175:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2419.346969] LustreError: 91206:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2419.365131] LustreError: 91206:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2419.376979] LustreError: 91206:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 2419.953646] Lustre: server umount lustre-MDT0000 complete [ 2436.661689] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2440.613542] Key type lgssc unregistered [ 2441.005864] LNet: 92707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2441.019359] LNetError: 92707:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2441.032724] LNet: Removed LNI 192.168.204.135@tcp [ 2441.990762] Key type .llcrypt unregistered [ 2441.997645] Key type ._llcrypt unregistered [ 2462.386849] Key type ._llcrypt registered [ 2462.389414] Key type .llcrypt registered [ 2462.523821] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2463.756900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2463.778658] alg: No test for adler32 (adler32-zlib) [ 2464.940546] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 2465.258268] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2466.976199] Key type lgssc registered [ 2468.433564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2477.227508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2477.247881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2479.000067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2484.451374] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2488.302538] LustreError: 93928:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2488.335662] LustreError: 93928:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2493.417758] LustreError: 93929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2494.694191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2495.427310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2500.803056] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2509.210612] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2516.697192] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2526.444296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2527.605867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2535.098500] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2543.980691] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2609.116299] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 15:55:01 (1782244501) [ 2611.381698] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2613.992399] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 15:55:06 (1782244506) [ 2619.881551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2619.890779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2619.911074] Lustre: Skipped 1 previous similar message [ 2619.922239] Lustre: Skipped 2 previous similar messages [ 2624.993499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2625.005984] Lustre: Skipped 2 previous similar messages [ 2625.901574] Lustre: server umount lustre-MDT0000 complete [ 2629.804321] LustreError: 93909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782244523 with bad export cookie 7956545595577722290 [ 2629.814102] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2629.818062] LustreError: 93909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2630.122931] LustreError: 93923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2630.220880] Lustre: server umount lustre-MDT0001 complete [ 2645.350225] Lustre: server umount lustre-OST0000 complete [ 2660.112732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2660.764427] LustreError: 98247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2661.015201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2666.465431] LustreError: 98248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2666.475719] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2675.700824] LustreError: 98248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2675.716367] LustreError: 98248:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2675.849238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2676.567233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2681.486743] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2689.653566] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2698.619977] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2713.193485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2713.390193] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2713.396146] LustreError: 98235:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2713.417989] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2713.425636] LustreError: 100329:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2713.430728] LustreError: 100329:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2713.436525] LustreError: 100329:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2713.486668] Lustre: server umount lustre-OST0000 complete [ 2713.489031] LustreError: 100329:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2723.255215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2723.464556] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2723.479600] LustreError: 98236:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2723.508229] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2723.517300] LustreError: 100796:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2723.523972] LustreError: 100796:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2723.527380] LustreError: 100796:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2723.640728] Lustre: server umount lustre-OST0000 complete [ 2723.649747] LustreError: 100796:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2734.202612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2734.374121] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2734.379217] Lustre: Found index 0 for lustre-OST0000, updating log [ 2734.384110] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2734.469966] Lustre: lustre-OST0000: new disk, initializing [ 2734.478110] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2734.484903] Lustre: 101333:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2734.755166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2741.812339] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2746.853597] Lustre: Failing over lustre-OST0000 [ 2747.255381] Lustre: server umount lustre-OST0000 complete [ 2753.079978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2753.092451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2753.103668] Lustre: Skipped 2 previous similar messages [ 2753.130220] Lustre: Skipped 1 previous similar message [ 2758.120133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2758.145519] Lustre: Skipped 1 previous similar message [ 2759.433084] Lustre: server umount lustre-MDT0000 complete [ 2763.233593] LustreError: 98248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2763.262450] LustreError: 98248:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2763.455461] LustreError: 98227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782244657 with bad export cookie 7956545595577725034 [ 2763.460417] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2763.468602] LustreError: 98227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2774.159422] Lustre: server umount lustre-MDT0001 complete [ 2783.572645] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 15:57:56 (1782244676) [ 2804.589905] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2818.332702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2818.749191] LustreError: 103681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2818.902184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2824.332662] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2834.015071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2839.726983] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2843.139370] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2850.733973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2851.697475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2851.706911] Lustre: Skipped 1 previous similar message [ 2851.717500] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2858.513360] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2859.880296] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2859.911496] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2859.912737] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2868.775749] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2868.902219] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2869.006800] Lustre: lustre-OST0001: new disk, initializing [ 2869.013616] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2869.022468] Lustre: 106001:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2877.297616] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2877.579122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2877.601991] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2877.750160] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2886.154110] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2897.408329] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2903.531475] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2903.539730] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2903.549591] Lustre: Skipped 1 previous similar message [ 2903.557918] Lustre: Skipped 2 previous similar messages [ 2913.762660] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2913.777214] Lustre: Skipped 7 previous similar messages [ 2917.344120] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2917.697070] Lustre: server umount lustre-MDT0001 complete [ 2918.892547] LustreError: 105198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2918.924988] LustreError: 105198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2928.964797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2929.152376] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2929.159100] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2929.228889] Lustre: 103672:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2930.186842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2930.196605] Lustre: Skipped 1 previous similar message [ 2930.709760] LustreError: 107544:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2930.719717] LustreError: 107544:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2930.727724] LustreError: 103674:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2930.896139] LustreError: 107551:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.135@tcp: cfg command failed: rc = -17 [ 2930.904887] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2930.904887] [ 2935.296559] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2935.315275] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2946.167887] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 16:00:39 (1782244839) [ 2964.308695] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2986.196589] Lustre: 109634:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3015.110784] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3019.437994] Lustre: 110769:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3113.955378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3113.964698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3113.997908] Lustre: Skipped 1 previous similar message [ 3114.008757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3114.019413] Lustre: Skipped 1 previous similar message [ 3119.151890] Lustre: server umount lustre-MDT0000 complete [ 3119.587486] LustreError: 106482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3119.621257] LustreError: 106482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 3122.922550] LustreError: 106000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782245016 with bad export cookie 7956545595577726035 [ 3122.924226] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3122.932668] LustreError: 106000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3123.109991] Lustre: server umount lustre-MDT0001 complete [ 3137.283779] Lustre: server umount lustre-OST0000 complete [ 3140.897118] Lustre: 93240:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782245018/real 1782245018] req@ffff9a784d604a80 x1868818661366272/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782245034 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3140.924846] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3140.933073] Lustre: Skipped 3 previous similar messages [ 3141.303285] Lustre: server umount lustre-OST0001 complete [ 3146.803154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3156.843775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3166.742441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3177.876576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3198.674899] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3210.275632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3210.301641] Lustre: lustre-MDT0000: reset Object Index mappings [ 3211.098973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3216.517856] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3226.647268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3226.678919] Lustre: lustre-MDT0001: reset Object Index mappings [ 3232.131031] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3234.927392] Lustre: 115866:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3243.219897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3252.111738] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3253.000838] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3256.069920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 3263.713494] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3269.670047] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3269.674639] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3271.303984] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3279.263725] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3282.863777] Lustre: 117746:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3344.111660] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 16:07:17 (1782245237) [ 3370.466050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3370.485537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3370.508227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3370.526501] Lustre: Skipped 4 previous similar messages [ 3376.683134] Lustre: server umount lustre-MDT0000 complete [ 3377.129163] LustreError: 114718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3377.161621] LustreError: 114718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 3381.457021] LustreError: 115868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782245275 with bad export cookie 7956545595577772886 [ 3381.458284] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3381.483983] LustreError: 115868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3388.146884] Lustre: server umount lustre-MDT0001 complete [ 3399.326239] Lustre: server umount lustre-OST0000 complete [ 3409.677314] Lustre: server umount lustre-OST0001 complete [ 3417.300833] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 3431.572924] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3473.527535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3473.828914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3473.887682] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3474.016092] Lustre: lustre-MDT0000: new disk, initializing [ 3474.120635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3474.137727] Lustre: Skipped 3 previous similar messages [ 3474.169712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3479.985652] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3488.470780] Lustre: server umount lustre-MDT0000 complete [ 3500.405765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3500.792716] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3500.826653] Lustre: 122037:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3500.844432] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3507.513304] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3513.796758] Lustre: server umount lustre-MDT0000 complete [ 3524.560496] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 16:10:17 (1782245417) [ 3526.173527] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3528.645844] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 16:10:21 (1782245421) [ 3546.488965] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3557.092877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3563.260792] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3574.070424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3574.177229] Lustre: 124088:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3574.238719] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3574.243361] Lustre: Skipped 1 previous similar message [ 3574.311774] Lustre: lustre-MDT0001: new disk, initializing [ 3574.536713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3574.560528] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3579.751258] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3585.215747] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3592.950335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3593.316900] Lustre: lustre-OST0000: new disk, initializing [ 3593.322429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3593.329815] Lustre: 125669:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3595.076291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3595.084843] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3595.235175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3600.529567] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3611.839314] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3612.081551] Lustre: lustre-OST0001: new disk, initializing [ 3612.084902] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3612.090944] Lustre: 126540:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3613.809418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3613.833566] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3614.001083] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3619.490985] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3631.402511] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3635.669871] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4070.091918] Lustre: 132962:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4070.098388] Lustre: 132962:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4070.105492] Lustre: 132962:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4070.114869] Lustre: 132962:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4070.122614] Lustre: 132962:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4070.128671] Lustre: 132962:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4401.695958] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:24:53 (1782246293) [ 4418.027238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4418.031479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4418.043329] Lustre: Skipped 8 previous similar messages [ 4418.053200] Lustre: Skipped 10 previous similar messages [ 4428.264968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4428.282020] Lustre: Skipped 8 previous similar messages [ 4430.304204] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4430.642600] Lustre: server umount lustre-MDT0000 complete [ 4433.388191] LustreError: 124095:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4433.412160] LustreError: 124095:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4433.956620] LustreError: 133017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782246327 with bad export cookie 7956545595577843817 [ 4433.957495] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4433.974090] LustreError: 133017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4438.499543] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4440.416317] Lustre: server umount lustre-MDT0001 complete [ 4450.337090] Lustre: server umount lustre-OST0000 complete [ 4459.559189] Lustre: server umount lustre-OST0001 complete [ 4468.025937] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 4478.041429] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 4516.638310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4516.976901] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4517.008233] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4517.095972] Lustre: lustre-MDT0000: new disk, initializing [ 4517.215314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4517.220368] Lustre: Skipped 5 previous similar messages [ 4517.249950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4521.511618] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4532.566763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4532.739716] Lustre: 136026:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4532.839831] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4532.849417] Lustre: Skipped 1 previous similar message [ 4533.006744] Lustre: lustre-MDT0001: new disk, initializing [ 4533.195043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4533.204575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4537.587884] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4546.841913] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4552.555813] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4561.548377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4561.831592] Lustre: lustre-OST0000: new disk, initializing [ 4561.838540] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4561.845178] Lustre: 137995:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4563.488820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4563.493613] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4563.599173] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4569.312844] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4578.666607] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4588.346486] Lustre: lustre-MDD0000: changelog on [ 4591.084495] Lustre: lustre-MDD0001: changelog on [ 6875.328830] Lustre: lustre-MDD0001: changelog off [ 6877.385383] Lustre: 136034:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6878.126817] Lustre: lustre-MDD0000: changelog off [ 6885.874713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6885.879970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6885.890915] Lustre: Skipped 2 previous similar messages [ 6885.904390] Lustre: Skipped 3 previous similar messages [ 6890.994404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6891.000093] Lustre: Skipped 1 previous similar message [ 6896.104545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6896.124057] Lustre: Skipped 1 previous similar message [ 6899.168183] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6899.439694] Lustre: server umount lustre-OST0000 complete [ 6903.778693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6903.785113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6906.337357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6906.339900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6906.345180] Lustre: Skipped 1 previous similar message [ 6909.870676] Lustre: server umount lustre-MDT0000 complete [ 6911.457304] LustreError: 136039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6911.476785] LustreError: 136039:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6913.734375] LustreError: 136020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782248807 with bad export cookie 7956545595578662222 [ 6913.737238] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6913.743299] LustreError: 136020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6914.162663] Lustre: server umount lustre-MDT0001 complete [ 6927.874377] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 6930.837679] Key type lgssc unregistered [ 6931.142725] LNet: 147217:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6931.153436] LNetError: 147217:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6931.174108] LNet: Removed LNI 192.168.204.135@tcp [ 6932.235221] Key type .llcrypt unregistered [ 6932.237435] Key type ._llcrypt unregistered [ 6953.938952] Key type ._llcrypt registered [ 6953.942448] Key type .llcrypt registered [ 6954.087894] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 6970.996518] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 6972.048716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6972.147299] alg: No test for adler32 (adler32-zlib) [ 6973.160761] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 6973.377667] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 6975.040323] Key type lgssc registered [ 6976.115324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7014.906653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7014.934194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7016.196408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7016.222677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7016.303134] Lustre: lustre-MDT0000: new disk, initializing [ 7016.383432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7016.401958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7020.411484] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7030.521768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7030.627153] Lustre: 150423:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7030.683164] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7030.686131] Lustre: Skipped 1 previous similar message [ 7030.794445] Lustre: lustre-MDT0001: new disk, initializing [ 7030.900737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7030.938728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7030.953443] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7035.361501] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7045.149209] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7051.068128] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7059.768185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7060.293735] Lustre: lustre-OST0000: new disk, initializing [ 7060.300979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7060.316136] Lustre: 152393:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7060.553341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7067.180834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7067.200834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7067.375705] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7068.663573] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7078.901436] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7084.982664] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_module obdecho/obdecho [ 7087.028263] LustreError: 153396:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7087.038862] LustreError: 153396:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7088.915505] LustreError: 153492:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 7102.435246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7102.445082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7102.473348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7103.465478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7103.468606] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7108.763283] Lustre: server umount lustre-MDT0000 complete [ 7113.020751] LustreError: 150415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782249007 with bad export cookie 18040717847001565796 [ 7113.025423] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7113.045257] LustreError: 150415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7113.504720] Lustre: server umount lustre-MDT0001 complete [ 7128.697649] Lustre: server umount lustre-OST0000 complete [ 7150.920900] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 7163.311890] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 7203.549118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7203.748801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7203.774797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7203.829411] Lustre: lustre-MDT0000: new disk, initializing [ 7203.883378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7203.897373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7208.495965] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7220.248788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7220.458722] Lustre: 157208:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7220.575036] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7220.585841] Lustre: Skipped 1 previous similar message [ 7220.735350] Lustre: lustre-MDT0001: new disk, initializing [ 7220.888713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7220.991397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7221.010750] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7229.108193] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7240.347503] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7246.274409] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7253.355765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7253.560680] Lustre: lustre-OST0000: new disk, initializing [ 7253.565452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7253.571118] Lustre: 159178:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7253.649427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7259.063381] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7262.248438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7262.254103] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7262.331668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7267.855380] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7273.308223] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7273.589937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7278.749651] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7279.012281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7282.657657] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7282.667129] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7282.677407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7282.693574] Lustre: Skipped 2 previous similar messages [ 7282.699383] Lustre: Skipped 2 previous similar messages [ 7287.723890] Lustre: server umount lustre-OST0000 complete [ 7292.386752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7292.397908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7292.432502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7292.440834] Lustre: Skipped 1 previous similar message [ 7298.019496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7298.041348] Lustre: Skipped 2 previous similar messages [ 7306.209631] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7306.404893] Lustre: server umount lustre-MDT0000 complete [ 7307.250936] LustreError: 157217:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7307.271538] LustreError: 157217:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7310.420342] LustreError: 157202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782249204 with bad export cookie 18040717847001566755 [ 7310.427779] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7310.433649] LustreError: 157202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7310.962941] Lustre: server umount lustre-MDT0001 complete [ 7314.810150] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 17:13:27 (1782249207) [ 7316.570557] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7318.739521] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 17:13:31 (1782249211) [ 7326.779800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7327.302836] LustreError: 161591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7327.377911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7331.971894] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7332.836678] LustreError: 161592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7337.956048] LustreError: 161591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7340.459795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7344.872479] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7351.852262] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7358.490409] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7365.252891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7365.618758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7365.621930] Lustre: Skipped 1 previous similar message [ 7373.932829] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7382.452165] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7392.227847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7392.234617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7392.248017] Lustre: Skipped 1 previous similar message [ 7392.253565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7392.256978] Lustre: Skipped 2 previous similar messages [ 7395.516405] Lustre: server umount lustre-MDT0000 complete [ 7398.381110] LustreError: 161592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7398.396323] LustreError: 161592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7401.044291] LustreError: 161571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782249295 with bad export cookie 18040717847001567637 [ 7401.051398] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7401.066446] LustreError: 161571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7403.491257] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7403.522614] Lustre: Skipped 2 previous similar messages [ 7406.498222] LustreError: 163025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7406.515214] LustreError: 163025:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7407.757313] Lustre: server umount lustre-MDT0001 complete [ 7416.462700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7430.234235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7430.474802] LustreError: 165434:0:(llog.c:1646:llog_backup()) MGC192.168.204.135@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7430.484447] Lustre: 165434:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.135@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7431.137650] LustreError: 165439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7464.970448] LustreError: 165459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7464.993263] LustreError: 165459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 7491.040340] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7491.045320] Lustre: 165434:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.135@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7530.506537] LustreError: 165441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7530.516031] LustreError: 165441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 41 previous similar messages [ 7532.019395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7537.148783] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7537.855762] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7547.909064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7556.581680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7571.937970] LustreError: 148642:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fa7fbfa8 x1868823388101632/t0(0) o250->MGC192.168.204.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7576.956961] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7577.594055] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7585.376749] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7587.851285] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7594.369790] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7596.450568] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7623.655467] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7623.658165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7623.676262] Lustre: Skipped 1 previous similar message [ 7623.700850] Lustre: Skipped 9 previous similar messages [ 7624.450220] Lustre: server umount lustre-MDT0000 complete [ 7628.656234] LustreError: 161573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782249522 with bad export cookie 18040717847001574126 [ 7628.661186] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7628.674946] LustreError: 161573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7629.001526] Lustre: server umount lustre-MDT0001 complete [ 7639.016622] Lustre: server umount lustre-OST0000 complete [ 7662.952675] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 7674.738982] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 7719.719987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7720.024220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7720.078397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7720.187649] Lustre: lustre-MDT0000: new disk, initializing [ 7720.304536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7720.307397] Lustre: Skipped 1 previous similar message [ 7720.327575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7725.134258] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7738.996409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7739.107714] Lustre: 170545:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7739.451463] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7744.770880] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7757.973839] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7765.587536] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7773.782556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7773.971411] Lustre: lustre-OST0000: new disk, initializing [ 7773.976747] Lustre: Skipped 1 previous similar message [ 7773.980701] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7773.988189] Lustre: Skipped 2 previous similar messages [ 7773.994300] Lustre: 172515:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7780.378594] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7781.983512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7782.000248] Lustre: Skipped 1 previous similar message [ 7782.014497] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7782.131123] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7791.755199] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7796.683827] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7796.858835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7802.020381] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7802.281820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7807.974804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7807.983241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7807.989210] LustreError: Skipped 1 previous similar message [ 7807.991356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7808.011470] Lustre: Skipped 3 previous similar messages [ 7808.028709] Lustre: Skipped 2 previous similar messages [ 7810.665566] Lustre: server umount lustre-OST0000 complete [ 7821.903746] Lustre: server umount lustre-MDT0000 complete [ 7823.348386] LustreError: 170557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7823.377276] LustreError: 170557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 39 previous similar messages [ 7826.104500] LustreError: 170537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782249720 with bad export cookie 18040717847001590926 [ 7826.113224] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7826.114757] LustreError: 170537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7826.475623] Lustre: server umount lustre-MDT0001 complete [ 7836.649598] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 17:22:09 (1782249729) [ 7844.437202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7850.176093] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7859.764454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7864.907182] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7873.573311] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7880.706084] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7888.458527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7895.244159] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7904.427680] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7919.554734] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 17:23:32 (1782249812) [ 7925.735206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7925.737499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7925.755887] Lustre: Skipped 2 previous similar messages [ 7925.770896] LustreError: Skipped 1 previous similar message [ 7925.785402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7925.804298] Lustre: Skipped 4 previous similar messages [ 7931.090441] Lustre: server umount lustre-OST0000 complete [ 7941.797819] Lustre: server umount lustre-MDT0000 complete [ 7945.928747] LustreError: 174907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782249839 with bad export cookie 18040717847001591808 [ 7945.932616] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7945.961278] LustreError: 174907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7946.346877] Lustre: server umount lustre-MDT0001 complete [ 7958.802834] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 7961.691741] Key type lgssc unregistered [ 7962.121769] LNet: 178661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7962.128167] LNetError: 178661:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7962.152948] LNet: Removed LNI 192.168.204.135@tcp [ 7963.034598] Key type .llcrypt unregistered [ 7963.037672] Key type ._llcrypt unregistered [ 7983.727728] Key type ._llcrypt registered [ 7983.730656] Key type .llcrypt registered [ 7983.861647] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 7985.298820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7985.339325] alg: No test for adler32 (adler32-zlib) [ 7986.589211] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 7986.871928] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 7988.675497] Key type lgssc registered [ 7989.922757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7998.842215] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7998.864992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8014.498447] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8019.616334] LustreError: 179904:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 8024.736815] LustreError: 179904:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8024.745604] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8024.750429] LustreError: MGC192.168.204.135@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8024.759390] LustreError: 179904:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8024.765771] LustreError: 179904:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 8024.771509] LustreError: 179904:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8024.842728] Lustre: server umount lustre-OST0000 complete [ 8024.846472] LustreError: 179904:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8034.411389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8035.894606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8040.761638] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8045.038125] LustreError: 180403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8045.058681] LustreError: 180403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8050.016162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8050.150355] LustreError: 180404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8050.496766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8055.370177] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8064.535814] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8071.525957] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8081.679621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8082.058872] LustreError: 182222:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8082.082113] Lustre: 182222:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.135@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8082.262878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8090.726286] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8099.616438] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8105.963906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8105.974880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8105.991128] Lustre: Skipped 2 previous similar messages [ 8106.002582] Lustre: Skipped 2 previous similar messages [ 8108.380236] Lustre: server umount lustre-MDT0000 complete [ 8111.074060] LustreError: 181406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8111.095991] LustreError: 181406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8112.750779] LustreError: 180383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782250006 with bad export cookie 10181855377684924234 [ 8112.752087] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8112.767030] LustreError: 180383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8113.274607] Lustre: server umount lustre-MDT0001 complete [ 8127.343479] Lustre: server umount lustre-OST0000 complete [ 8137.144690] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 8148.665820] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 8189.296667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8189.516608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8189.546557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8189.648875] Lustre: lustre-MDT0000: new disk, initializing [ 8189.735235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8189.752209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8194.375225] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8204.575801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8204.711532] Lustre: 185749:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8204.790692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8204.795033] Lustre: Skipped 1 previous similar message [ 8204.900860] Lustre: lustre-MDT0001: new disk, initializing [ 8205.007586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8205.038427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8205.046085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8209.731692] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8218.126112] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8223.609180] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8231.026161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8231.351294] Lustre: lustre-OST0000: new disk, initializing [ 8231.355538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8231.367458] Lustre: 187718:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8231.473933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8237.644753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8237.659789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8237.828882] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8238.058629] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8246.802091] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8251.295058] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8251.507511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8256.440531] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8256.693323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8258.541719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8258.542369] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8258.551544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8258.552024] Lustre: Skipped 1 previous similar message [ 8258.557660] LustreError: Skipped 1 previous similar message [ 8263.661520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8263.668143] Lustre: Skipped 2 previous similar messages [ 8264.842607] Lustre: server umount lustre-OST0000 complete [ 8271.849759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8271.856917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8271.895958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8273.898315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8275.511281] Lustre: server umount lustre-MDT0000 complete [ 8279.009402] LustreError: 185761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8279.033428] LustreError: 185761:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8279.496832] LustreError: 185740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782250173 with bad export cookie 10181855377684924948 [ 8279.501847] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8279.508498] LustreError: 185740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8279.970469] Lustre: server umount lustre-MDT0001 complete [ 8292.037196] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 17:29:44 (1782250184) [ 8312.524505] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 8314.721789] Key type lgssc unregistered [ 8314.962558] LNet: 190666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8314.966835] LNetError: 190666:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8314.979479] LNet: Removed LNI 192.168.204.135@tcp [ 8315.576914] Key type .llcrypt unregistered [ 8315.579700] Key type ._llcrypt unregistered [ 8333.552903] Key type ._llcrypt registered [ 8333.558818] Key type .llcrypt registered [ 8333.661813] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 8334.319259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8334.359613] alg: No test for adler32 (adler32-zlib) [ 8335.603313] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 8335.782851] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 8337.424280] Key type lgssc registered [ 8338.538624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8347.229433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8347.253723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8348.679261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8354.090828] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8358.896476] LustreError: 191883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8358.917775] LustreError: 191883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8364.013203] LustreError: 191884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8364.185490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8364.485889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8369.961383] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8379.026407] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8389.558558] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8401.280376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8401.429278] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8401.432234] LustreError: 193750:0:(llog.c:1687:llog_backup()) MGC192.168.204.135@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8401.438797] Lustre: 193750:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.135@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8401.595710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8408.310303] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8415.298405] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8417.980438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8417.993411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8417.995687] Lustre: Skipped 2 previous similar messages [ 8418.016793] Lustre: Skipped 2 previous similar messages [ 8423.399971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8423.404888] Lustre: Skipped 2 previous similar messages [ 8424.610582] Lustre: server umount lustre-MDT0000 complete [ 8428.514904] LustreError: 191883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8428.535101] LustreError: 191883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8430.533227] LustreError: 191863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782250324 with bad export cookie 12821207599203925583 [ 8430.535221] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8430.541444] LustreError: 191863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8431.029640] Lustre: server umount lustre-MDT0001 complete [ 8446.069978] Lustre: server umount lustre-OST0000 complete [ 8455.752404] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 8466.832664] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 8509.079127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8509.372243] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8509.425302] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8509.509905] Lustre: lustre-MDT0000: new disk, initializing [ 8509.576385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8509.593256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8514.175528] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8525.646586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8525.802153] Lustre: 197279:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8525.891910] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8525.896349] Lustre: Skipped 1 previous similar message [ 8525.903327] LustreError: 198061:0:(llog.c:1687:llog_backup()) MGC192.168.204.135@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8525.920643] LustreError: 198061:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 8525.930592] Lustre: 198061:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.135@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8525.984584] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8525.991476] Lustre: Skipped 1 previous similar message [ 8526.060508] Lustre: lustre-MDT0001: new disk, initializing [ 8526.170405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8526.204461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8526.213619] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8530.927751] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8541.177677] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8546.972989] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8553.929197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8554.210687] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8554.213532] Lustre: Skipped 1 previous similar message [ 8554.216483] LustreError: 199222:0:(llog.c:1687:llog_backup()) MGC192.168.204.135@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8554.222596] LustreError: 199222:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 8554.231984] Lustre: 199222:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.135@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8554.239238] Lustre: 199222:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8554.264706] Lustre: lustre-OST0000: new disk, initializing [ 8554.268757] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8554.278723] Lustre: 199247:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8554.358114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8560.199266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8560.211363] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8560.312413] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8560.496979] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8569.995648] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8574.686445] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8574.938321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8580.103601] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8580.374071] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8586.210957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8586.214807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8586.222689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8586.232723] Lustre: Skipped 1 previous similar message [ 8586.244125] Lustre: Skipped 1 previous similar message [ 8588.527397] Lustre: server umount lustre-OST0000 complete [ 8592.869334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8592.888090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8592.900314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8596.456959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8598.516374] Lustre: server umount lustre-MDT0000 complete [ 8601.569359] LustreError: 197285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8601.594405] LustreError: 197285:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8602.065108] LustreError: 197271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782250496 with bad export cookie 12821207599203926297 [ 8602.067747] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8602.076921] LustreError: 197271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8602.451630] Lustre: server umount lustre-MDT0001 complete [ 8612.493608] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 17:35:05 (1782250505) [ 8628.465614] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 8637.946801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8638.310101] LustreError: 202374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8638.352132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8642.166105] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8648.675855] LustreError: 202374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8648.709633] LustreError: 202374:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8651.075836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8656.497967] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8659.450176] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8666.284907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8672.978995] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8682.169877] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8682.376158] Lustre: lustre-OST0001: new disk, initializing [ 8682.379928] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8682.385522] Lustre: 204682:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8682.492316] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8682.496717] Lustre: Skipped 2 previous similar messages [ 8688.704429] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8692.371549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8692.390668] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8692.447220] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8699.087629] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8710.198304] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8712.494819] Lustre: DEBUG MARKER: ADD OST3 [ 8718.349509] loop: module loaded [ 8718.416306] loop0: detected capacity change from 0 to 204800000 [ 8718.635218] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8718.651727] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8718.668359] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8725.316708] loop0: detected capacity change from 0 to 204800000 [ 8726.105396] Lustre: DEBUG MARKER: STOP OST3 [ 8732.017538] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8732.230846] Lustre: lustre-OST0003: new disk, initializing [ 8732.235426] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8732.242946] Lustre: 206698:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8734.004525] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8734.171080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8734.179792] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8734.232111] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8735.882185] Lustre: Failing over lustre-OST0003 [ 8736.033420] LustreError: 206866:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8736.077389] Lustre: server umount lustre-OST0003 complete [ 8738.892632] LustreError: 203860:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8738.955039] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8739.309181] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8747.609742] Lustre: DEBUG MARKER: START OST3 again [ 8753.998793] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8754.280936] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8754.292778] Lustre: Skipped 1 previous similar message [ 8754.311488] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8755.503639] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8756.036965] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8756.038981] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8760.874087] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8781.026067] Lustre: Failing over lustre-OST0003 [ 8781.142954] Lustre: server umount lustre-OST0003 complete [ 8781.795537] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8781.812417] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8781.831586] Lustre: Skipped 1 previous similar message [ 8781.845260] LustreError: 203858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8781.862427] LustreError: 203858:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 8799.713585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8799.722632] LustreError: Skipped 1 previous similar message [ 8799.727292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8799.739887] Lustre: Skipped 1 previous similar message [ 8799.757482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8799.765077] Lustre: Skipped 2 previous similar messages [ 8805.949850] Lustre: server umount lustre-MDT0000 complete [ 8810.193530] LustreError: 204680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782250704 with bad export cookie 12821207599203927179 [ 8810.209440] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8810.212231] LustreError: 204680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8810.477729] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8810.488180] Lustre: Skipped 8 previous similar messages [ 8816.709446] Lustre: server umount lustre-MDT0001 complete [ 8821.046268] Lustre: server umount lustre-OST0000 complete [ 8825.160829] Lustre: server umount lustre-OST0001 complete [ 8832.525807] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 8841.256369] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 8882.784932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8883.036243] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8883.084486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8883.181260] Lustre: lustre-MDT0000: new disk, initializing [ 8883.277210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8883.294772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8888.065436] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8899.778186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8899.914579] Lustre: 212143:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8900.073817] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8900.079885] Lustre: Skipped 1 previous similar message [ 8900.175134] Lustre: lustre-MDT0001: new disk, initializing [ 8900.331472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8900.344723] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8905.055479] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8914.912252] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8921.095883] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8929.030941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8929.495168] Lustre: 214114:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8930.703450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8930.850922] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8936.609898] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8946.416348] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8951.526549] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8951.686219] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8955.986594] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8956.294497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8961.520136] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8961.525855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8961.538779] Lustre: Skipped 5 previous similar messages [ 8961.544450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8961.547409] Lustre: Skipped 2 previous similar messages [ 8965.402363] Lustre: server umount lustre-OST0000 complete [ 8977.174355] Lustre: server umount lustre-MDT0000 complete [ 8981.296861] LustreError: 212136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782250875 with bad export cookie 12821207599203929475 [ 8981.304254] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8981.305245] LustreError: 212136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8981.682485] Lustre: server umount lustre-MDT0001 complete [ 8985.534123] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 17:41:18 (1782250878) [ 9002.715259] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 9012.968294] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 9057.433529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9057.811192] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9057.864647] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9057.878071] Lustre: Skipped 1 previous similar message [ 9058.058460] Lustre: lustre-MDT0000: new disk, initializing [ 9058.065054] Lustre: Skipped 1 previous similar message [ 9058.226122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9058.235609] Lustre: Skipped 1 previous similar message [ 9063.809209] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9076.433443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9076.647305] Lustre: 218726:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9077.224742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9084.171611] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9096.232890] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9103.408707] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9111.745285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9112.281751] Lustre: 220697:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9113.886056] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9114.027760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9120.194148] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9130.165206] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9135.939933] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9136.174651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9141.522946] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9141.827843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9144.805459] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9144.805886] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9144.816135] LustreError: Skipped 1 previous similar message [ 9144.843174] Lustre: Skipped 3 previous similar messages [ 9144.847119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9144.857433] Lustre: Skipped 4 previous similar messages [ 9158.112345] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9158.499660] Lustre: server umount lustre-OST0000 complete [ 9164.257029] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9164.268307] LustreError: Skipped 1 previous similar message [ 9169.051206] Lustre: server umount lustre-MDT0000 complete [ 9170.403508] LustreError: 220557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9170.427370] LustreError: 220557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 9173.132061] LustreError: 218718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782251067 with bad export cookie 12821207599203930357 [ 9173.149090] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9173.152138] LustreError: 218718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9173.583021] Lustre: server umount lustre-MDT0001 complete [ 9182.491680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9183.000791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9183.008278] Lustre: Skipped 5 previous similar messages [ 9187.855771] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9195.436852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9199.721939] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9207.248252] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9212.997873] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9220.096086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9227.221845] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9235.722377] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9247.201131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9247.213910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9247.219093] Lustre: Skipped 7 previous similar messages [ 9252.282368] Lustre: server umount lustre-OST0000 complete [ 9262.921656] Lustre: server umount lustre-MDT0000 complete [ 9266.900757] LustreError: 222896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782251160 with bad export cookie 12821207599203931239 [ 9266.907871] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9266.916431] LustreError: 222896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9267.287421] Lustre: server umount lustre-MDT0001 complete [ 9377.297303] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 9379.865176] Key type lgssc unregistered [ 9380.188422] LNet: 226586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9380.201881] LNetError: 226586:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9380.221091] LNet: Removed LNI 192.168.204.135@tcp [ 9381.129286] Key type .llcrypt unregistered [ 9381.136312] Key type ._llcrypt unregistered [ 9409.829409] Key type ._llcrypt registered [ 9409.835282] Key type .llcrypt registered [ 9409.951273] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 9428.664083] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 9429.774452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9429.895961] alg: No test for adler32 (adler32-zlib) [ 9430.987228] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 9431.250617] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 9433.000213] Key type lgssc registered [ 9434.477962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9474.014404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9474.043963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9475.255870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9475.285430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9475.367424] Lustre: lustre-MDT0000: new disk, initializing [ 9475.450600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9475.468742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9480.096380] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9491.892683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9491.996670] Lustre: 229941:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9492.036297] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9492.042074] Lustre: Skipped 1 previous similar message [ 9492.089698] Lustre: lustre-MDT0001: new disk, initializing [ 9492.200545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9492.271560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9492.291336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9496.969943] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9508.933477] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9516.529294] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9524.353632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9524.652424] Lustre: lustre-OST0000: new disk, initializing [ 9524.662915] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9524.674786] Lustre: 231911:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9524.753527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9530.480156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9530.494124] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9530.673064] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9531.544116] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9541.815481] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9547.032164] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9547.293388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9552.008908] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9552.236628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9556.451591] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9556.455211] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9556.463453] LustreError: Skipped 1 previous similar message [ 9556.477513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9556.485643] Lustre: Skipped 1 previous similar message [ 9561.576561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9561.585093] Lustre: Skipped 1 previous similar message [ 9566.698467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9566.708845] Lustre: Skipped 1 previous similar message [ 9568.224148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9568.552329] Lustre: server umount lustre-OST0000 complete [ 9574.370935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9574.388514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9574.399685] Lustre: Skipped 1 previous similar message [ 9574.402940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9576.931903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9578.745154] Lustre: server umount lustre-MDT0000 complete [ 9582.211961] LustreError: 229935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782251476 with bad export cookie 11350598518284583970 [ 9582.217866] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9582.222746] LustreError: 229935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9582.528753] Lustre: server umount lustre-MDT0001 complete [ 9603.474702] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 9617.128246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9617.744513] LustreError: 234849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9617.771247] LustreError: 234849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9617.861914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9623.019887] LustreError: 234850:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9623.708926] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9627.113937] LustreError: 234849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9632.235196] LustreError: 234850:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9633.915610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9634.380480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9639.448591] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9642.644163] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9649.526114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9649.877855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9656.412440] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9664.979925] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9665.136932] Lustre: lustre-OST0001: new disk, initializing [ 9665.143165] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9665.155373] Lustre: 237157:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9671.248958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9671.263061] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9671.351891] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9671.525693] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9682.629676] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9694.380476] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9707.368884] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 17:53:20 (1782251600) [ 9711.075075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9711.082029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9711.098972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9711.113818] Lustre: Skipped 2 previous similar messages [ 9722.853911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9722.856435] Lustre: Skipped 9 previous similar messages [ 9723.360118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9723.599165] Lustre: server umount lustre-MDT0000 complete [ 9727.465939] LustreError: 237155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782251621 with bad export cookie 11350598518284584845 [ 9727.472885] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9727.482536] LustreError: 237155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9727.805552] Lustre: server umount lustre-MDT0001 complete [ 9741.841735] Lustre: server umount lustre-OST0000 complete [ 9743.312643] Lustre: 228167:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782251621/real 1782251621] req@ffff9a797bdf6680 x1868825965482112/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782251637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9743.359757] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9743.378450] Lustre: Skipped 3 previous similar messages [ 9745.536555] Lustre: server umount lustre-OST0001 complete [ 9751.886880] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 9763.971629] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 9799.231326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9799.513915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9799.554575] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9799.689216] Lustre: lustre-MDT0000: new disk, initializing [ 9799.799970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9799.816163] Lustre: Skipped 1 previous similar message [ 9799.861546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9804.875927] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9815.683597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9815.821974] Lustre: 241137:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9815.892272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9815.896942] Lustre: Skipped 1 previous similar message [ 9816.008574] Lustre: lustre-MDT0001: new disk, initializing [ 9816.202568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9816.217295] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9821.229612] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9830.377784] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9835.638740] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9842.076371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9842.368612] Lustre: lustre-OST0000: new disk, initializing [ 9842.372805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9842.379264] Lustre: 243072:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9842.516503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9842.522740] Lustre: Skipped 1 previous similar message [ 9843.485520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9843.503465] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9843.610070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9850.713037] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9860.380809] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9866.088302] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9866.324988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9871.758706] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9872.097979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9874.410626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9874.415587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9874.423014] LustreError: Skipped 1 previous similar message [ 9874.426021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9874.451517] Lustre: Skipped 1 previous similar message [ 9880.489461] Lustre: server umount lustre-OST0000 complete [ 9890.705973] Lustre: server umount lustre-MDT0000 complete [ 9894.781609] LustreError: 243071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782251788 with bad export cookie 11350598518284586987 [ 9894.783895] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9894.791511] LustreError: 243071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9895.157605] Lustre: server umount lustre-MDT0001 complete [ 9904.183420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9904.552415] LustreError: 245288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9909.007670] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9914.860293] LustreError: 245289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9914.876939] LustreError: 245289:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9917.373314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9917.846139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9917.853837] Lustre: Skipped 1 previous similar message [ 9922.366913] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9930.401501] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9936.177539] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9942.792924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9950.241626] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9958.562506] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10106.338228] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10106.353265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10106.370944] Lustre: Skipped 2 previous similar messages [10106.380527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10106.388090] Lustre: Skipped 8 previous similar messages [10111.328945] Lustre: server umount lustre-OST0000 complete [10129.377231] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10129.564857] Lustre: server umount lustre-MDT0000 complete [10131.937368] LustreError: 245284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10131.955687] LustreError: 245284:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10133.353925] LustreError: 245268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782252027 with bad export cookie 11350598518284587869 [10133.363122] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10133.364973] LustreError: 245268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10133.670923] Lustre: server umount lustre-MDT0001 complete [10229.239918] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [10232.101886] Key type lgssc unregistered [10232.425692] LNet: 248906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10232.433458] LNetError: 248906:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10232.446570] LNet: Removed LNI 192.168.204.135@tcp [10233.216151] Key type .llcrypt unregistered [10233.219544] Key type ._llcrypt unregistered [10257.231359] Key type ._llcrypt registered [10257.234659] Key type .llcrypt registered [10257.343780] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [10273.430840] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [10274.666550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10274.735744] alg: No test for adler32 (adler32-zlib) [10275.924156] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [10276.191174] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [10277.896629] Key type lgssc registered [10279.242235] Lustre: Echo OBD driver; http://www.lustre.org/ [10332.544429] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [10343.581371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10343.628460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10345.042749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10345.083510] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10345.166427] Lustre: lustre-MDT0000: new disk, initializing [10345.245530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10345.265755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10350.205140] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10361.781907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10361.956283] Lustre: 252992:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10362.016105] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10362.020766] Lustre: Skipped 1 previous similar message [10362.094336] Lustre: lustre-MDT0001: new disk, initializing [10362.238139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10362.355130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10362.378644] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10367.256832] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10372.419396] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10379.916535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10380.281277] Lustre: lustre-OST0000: new disk, initializing [10380.286791] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10380.312319] Lustre: 254625:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10380.450907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10386.512032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10386.525088] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10386.679899] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10386.985600] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10397.939687] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10398.107432] Lustre: lustre-OST0001: new disk, initializing [10398.109936] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10398.117709] Lustre: 255497:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10398.256451] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10405.721298] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10407.012910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10407.026622] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10407.129036] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10417.883474] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10424.289905] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10438.121617] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10438.124857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10438.135117] Lustre: Skipped 3 previous similar messages [10438.152173] Lustre: Skipped 3 previous similar messages [10443.132269] Lustre: server umount lustre-MDT0000 complete [10446.696213] LustreError: 252984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782252340 with bad export cookie 682622516821369754 [10446.703411] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10446.710461] LustreError: 252984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10447.144413] Lustre: server umount lustre-MDT0001 complete [10461.863775] Lustre: server umount lustre-OST0000 complete [10464.746025] Lustre: 250485:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782252342/real 1782252342] req@ffff9a7950b86680 x1868826851488768/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782252358 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10464.770972] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10465.852974] Lustre: server umount lustre-OST0001 complete [10480.946137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10491.984948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10502.045837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10513.823777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10535.906842] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [10549.564748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10549.599758] Lustre: lustre-MDT0000: reset Object Index mappings [10549.964744] LustreError: 260508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10549.975297] LustreError: 260508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10549.996521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10555.365669] LustreError: 260509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10556.137658] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10559.463337] LustreError: 260508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10564.580026] LustreError: 260509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10565.200073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10565.216724] Lustre: lustre-MDT0001: reset Object Index mappings [10565.679589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10569.896652] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10572.814277] Lustre: 261649:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10580.075388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10587.287167] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10590.644938] LustreError: 262004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10594.741686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10596.905930] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10597.327916] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10597.333695] Lustre: Skipped 1 previous similar message [10602.542873] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10605.593538] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10614.419046] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10618.829092] Lustre: 263522:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10625.186754] Lustre: server umount lustre-MDT0000 complete [10627.042283] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10627.062431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10627.096717] LustreError: 260508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10627.116885] LustreError: 260508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10629.850373] LustreError: 263524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782252523 with bad export cookie 682622516821376544 [10629.855948] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10629.862070] LustreError: 263524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10630.249364] Lustre: server umount lustre-MDT0001 complete [10644.424258] Lustre: server umount lustre-OST0000 complete [10658.310981] Lustre: server umount lustre-OST0001 complete [10679.214045] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 18:09:32 (1782252572) [10696.007934] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [10706.904065] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [10744.147699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10744.399994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10744.430100] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10744.520727] Lustre: lustre-MDT0000: new disk, initializing [10744.649075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10744.683395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10749.836398] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10761.974233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10762.152853] Lustre: 267778:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10762.250973] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10762.265133] Lustre: Skipped 1 previous similar message [10762.352083] Lustre: lustre-MDT0001: new disk, initializing [10762.574794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10762.599757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10767.683317] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10778.127210] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10785.505100] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10794.009952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10794.470168] Lustre: lustre-OST0000: new disk, initializing [10794.479075] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10794.510300] Lustre: 269749:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10796.020565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10796.042837] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10796.313181] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10802.637415] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10812.996291] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10818.620810] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10818.889275] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10824.143827] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10824.390062] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10826.722129] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10826.729226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10826.742487] Lustre: Skipped 3 previous similar messages [10826.749171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10830.313407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10830.318674] Lustre: Skipped 2 previous similar messages [10832.544096] Lustre: server umount lustre-OST0000 complete [10839.529594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10839.546877] LustreError: Skipped 1 previous similar message [10839.554475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10839.573435] Lustre: Skipped 1 previous similar message [10839.585504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10845.667868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10845.673889] Lustre: Skipped 3 previous similar messages [10851.296779] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10851.489475] Lustre: server umount lustre-MDT0000 complete [10855.354955] LustreError: 267771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782252749 with bad export cookie 682622516821378637 [10855.373502] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10855.373678] LustreError: 267771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10855.702168] Lustre: server umount lustre-MDT0001 complete [10877.558832] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [10889.470919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10890.062712] LustreError: 272685:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10890.090477] LustreError: 272685:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10890.167896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10890.185632] Lustre: Skipped 2 previous similar messages [10895.013987] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10904.220978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10909.715782] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10912.554450] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10919.448069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10926.360805] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10935.167188] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10935.403590] Lustre: lustre-OST0001: new disk, initializing [10935.406749] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10935.411319] Lustre: 274993:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10937.251171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10937.262503] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10937.369325] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10941.943407] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10951.803612] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10955.659206] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10961.466909] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10961.480775] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10968.040570] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10968.044230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10968.056548] Lustre: Skipped 4 previous similar messages [10968.063226] Lustre: Skipped 4 previous similar messages [10973.935398] Lustre: server umount lustre-MDT0000 complete [10977.705293] LustreError: 272665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782252871 with bad export cookie 682622516821379519 [10977.707641] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10977.715670] LustreError: 272665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10978.149641] Lustre: server umount lustre-MDT0001 complete [10991.336355] Lustre: server umount lustre-OST0000 complete [10994.656465] Lustre: 250485:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782252872/real 1782252872] req@ffff9a784363bb80 x1868826851649664/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782252888 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10994.706243] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10995.019236] Lustre: server umount lustre-OST0001 complete [11012.673732] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [11023.449838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11023.833455] LustreError: 278053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11023.858968] LustreError: 278053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11023.937329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11023.946199] Lustre: Skipped 3 previous similar messages [11029.160684] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11039.450139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11044.738287] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11048.302760] Lustre: 279193:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11055.319888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11061.714806] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11068.090240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11070.354893] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11076.110969] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11077.393886] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11085.233561] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11089.086767] Lustre: 281075:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11094.557198] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11094.575088] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11103.660287] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 18:16:37 (1782252997) [11122.147640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11122.160881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11122.167605] Lustre: Skipped 3 previous similar messages [11122.177904] Lustre: Skipped 7 previous similar messages [11124.425670] Lustre: server umount lustre-MDT0000 complete [11127.269996] LustreError: 279064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11127.284744] LustreError: 279064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [11128.661775] LustreError: 281682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782253022 with bad export cookie 682622516821386057 [11128.672777] LustreError: 281682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11128.673128] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11135.192935] Lustre: server umount lustre-MDT0001 complete [11153.377074] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11153.485358] Lustre: server umount lustre-OST0000 complete [11171.296544] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11171.604635] Lustre: server umount lustre-OST0001 complete [11179.418463] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [11189.202916] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [11231.915405] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [11240.253415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11240.490115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11240.526552] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11240.607640] Lustre: lustre-MDT0000: new disk, initializing [11240.710448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11244.203828] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11253.440166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11253.538129] Lustre: 285313:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11253.747655] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11257.186587] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11268.501116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11269.108310] Lustre: 286944:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11270.637802] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11276.804741] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11287.120194] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11287.293727] Lustre: 287815:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11287.410613] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11287.418504] Lustre: Skipped 6 previous similar messages [11288.619346] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11288.629925] Lustre: Skipped 1 previous similar message [11288.807104] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11293.142128] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11303.021469] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11306.775574] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11306.779663] Lustre: Skipped 1 previous similar message [11319.436246] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 18:20:12 (1782253212) [11324.897853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11324.904301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11324.918468] Lustre: Skipped 4 previous similar messages [11324.943220] Lustre: Skipped 7 previous similar messages [11327.647390] Lustre: server umount lustre-MDT0000 complete [11330.019178] LustreError: 285325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11330.033693] LustreError: 285325:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [11331.127967] LustreError: 286109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782253225 with bad export cookie 682622516821393722 [11331.152139] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11331.154979] LustreError: 286109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11331.387686] Lustre: server umount lustre-MDT0001 complete [11345.773776] Lustre: server umount lustre-OST0000 complete [11360.245646] Lustre: server umount lustre-OST0001 complete [11368.632426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11373.872878] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11384.291194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11389.831532] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11399.238451] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11405.972643] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11413.437472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11420.402529] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11429.378443] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11446.178143] loop1: detected capacity change from 0 to 204800000 [11446.401726] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11446.415191] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11446.425734] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11451.566454] loop1: detected capacity change from 0 to 204800000 [11457.497475] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11457.563075] LustreError: 290055:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs1997: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11457.573520] LustreError: 290055:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs1997 [11457.580916] LustreError: 290055:0:(mgs_handler.c:517:mgs_target_reg()) fs1997-MDT0001: Error checking target index: rc = -13 [11457.592232] LustreError: fs1997-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11457.598816] LustreError: 293786:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11457.604362] LustreError: 293786:0:(tgt_mount.c:2129:server_put_super()) no obd fs1997-MDT0001 [11457.607393] LustreError: 293786:0:(tgt_mount.c:116:server_deregister_mount()) fs1997-MDT0001 not registered [11457.669495] Lustre: server umount fs1997-MDT0001 complete [11457.679177] LustreError: 293786:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11465.653078] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11465.800310] Lustre: Setting parameter fs1997-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs1997-MDT0001 [11465.855155] Lustre: srv-fs1997-MDT0001: No data found on store. Initialize space. [11465.862863] Lustre: Skipped 4 previous similar messages [11465.950566] Lustre: fs1997-MDT0001: new disk, initializing [11465.955795] Lustre: Skipped 3 previous similar messages [11466.084414] Lustre: srv-fs1997-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11466.097311] Lustre: 294253:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1997-MDT0001-mdtlov: can't create new llog: rc = -115 [11466.106374] LustreError: 294253:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1997-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11470.377739] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11477.124601] Lustre: server umount fs1997-MDT0001 complete [11491.514846] loop1: detected capacity change from 0 to 204800000 [11495.918697] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11497.340295] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11497.702971] Lustre: 295820:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1997-MDT0001-mdtlov: can't create new llog: rc = -115 [11497.717419] LustreError: 295820:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1997-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11502.437123] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11507.297492] Lustre: server umount fs1997-MDT0001 complete [11511.276020] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11517.520935] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.35@tcp (stopping) [11517.926564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11517.954877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11517.964315] Lustre: Skipped 1 previous similar message [11523.339830] Lustre: server umount lustre-MDT0000 complete [11532.315360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11532.440472] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11537.904681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11537.905991] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11545.916482] loop1: detected capacity change from 0 to 204800000 [11551.226667] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11551.602407] Lustre: 298266:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1997-MDT0001-mdtlov: can't create new llog: rc = -115 [11551.616391] LustreError: 298266:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1997-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11556.167362] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11560.991185] Lustre: server umount fs1997-MDT0001 complete [11583.970502] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11589.103307] LustreError: 292673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11589.116625] LustreError: 292673:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 105 previous similar messages [11589.568627] Lustre: server umount lustre-OST0000 complete [11599.036357] Lustre: server umount lustre-MDT0000 complete [11602.808579] LustreError: 290046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782253496 with bad export cookie 682622516821396403 [11602.809878] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11602.825609] LustreError: 290046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11603.027074] Lustre: server umount lustre-MDT0001 complete [11614.638134] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [11617.211618] Key type lgssc unregistered [11617.457150] LNet: 300610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11617.461304] LNetError: 300610:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11617.480592] LNet: Removed LNI 192.168.204.135@tcp [11618.373444] Key type .llcrypt unregistered [11618.377175] Key type ._llcrypt unregistered [11638.890738] Key type ._llcrypt registered [11638.893270] Key type .llcrypt registered [11639.013481] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [11640.276269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11640.341362] alg: No test for adler32 (adler32-zlib) [11641.474844] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [11641.702413] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [11643.390272] Key type lgssc registered [11644.437689] Lustre: Echo OBD driver; http://www.lustre.org/ [11652.809353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11652.826861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11654.283417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11658.299726] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11664.870894] LustreError: 301884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11664.891917] LustreError: 301884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11666.078188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11666.418662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11670.542975] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11678.392569] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11684.415144] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11690.973481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11691.442574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11692.471677] LustreError: 303714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11697.649839] LustreError: 303714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11698.683957] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11699.688387] LustreError: 303717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11699.697558] LustreError: 303717:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11704.808810] LustreError: 303714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11704.832459] LustreError: 303714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11707.090497] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11714.593562] LustreError: 303717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11714.621662] LustreError: 303717:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11723.091539] loop1: detected capacity change from 0 to 204800000 [11723.201802] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11723.213510] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11723.231052] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11727.764411] loop1: detected capacity change from 0 to 204800000 [11732.475773] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11732.595730] Lustre: fs3732-OST0000: new disk, initializing [11732.606940] Lustre: srv-fs3732-OST0000: No data found on store. Initialize space. [11732.615686] Lustre: 305620:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11732.696533] Lustre: fs3732-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11735.055899] LustreError: 303717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11735.078535] LustreError: 303717:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [11738.814618] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11746.737803] Lustre: server umount fs3732-OST0000 complete [11756.278698] loop1: detected capacity change from 0 to 204800000 [11761.701993] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11761.958625] Lustre: fs3732-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11768.648116] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11770.890807] LustreError: 303767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11770.904135] LustreError: 303767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [11776.509172] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11778.308237] loop2: detected capacity change from 0 to 204800000 [11778.640501] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11778.666402] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11778.679182] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11781.626762] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11784.048454] loop2: detected capacity change from 0 to 204800000 [11789.296654] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11789.405101] LustreError: fs3732-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11789.415663] LustreError: fs3732-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11789.426316] LustreError: 308659:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11789.435818] LustreError: 308659:0:(tgt_mount.c:2129:server_put_super()) no obd fs3732-OST0001 [11789.442902] LustreError: 308659:0:(tgt_mount.c:116:server_deregister_mount()) fs3732-OST0001 not registered [11789.468439] Lustre: server umount fs3732-OST0001 complete [11789.471345] LustreError: 308659:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11799.907178] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11800.112875] Lustre: fs3732-OST0001: new disk, initializing [11800.119683] Lustre: srv-fs3732-OST0001: No data found on store. Initialize space. [11800.129041] Lustre: 309209:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11800.241215] Lustre: fs3732-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11806.033725] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11813.831034] Lustre: server umount fs3732-OST0000 complete [11822.538900] Lustre: server umount fs3732-OST0001 complete [11835.789584] loop1: detected capacity change from 0 to 204800000 [11837.460631] LustreError: 303767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11837.502238] LustreError: 303767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [11842.192950] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11842.446055] Lustre: fs3732-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11850.002416] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11857.547166] loop2: detected capacity change from 0 to 204800000 [11863.373315] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11871.993892] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11882.292860] Lustre: server umount fs3732-OST0000 complete [11891.487620] Lustre: server umount fs3732-OST0001 complete [11904.481522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11904.488943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11904.494201] Lustre: Skipped 1 previous similar message [11909.602135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11909.615884] Lustre: Skipped 2 previous similar messages [11914.735912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11914.742624] Lustre: Skipped 1 previous similar message [11917.287520] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11917.582958] Lustre: server umount lustre-OST0000 complete [11922.401482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11922.415760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11922.460291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11924.964227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11928.095467] Lustre: server umount lustre-MDT0000 complete [11932.024681] LustreError: 313127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782253826 with bad export cookie 6498277839344481084 [11932.032660] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11932.035069] LustreError: 313127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11932.554417] Lustre: server umount lustre-MDT0001 complete [11944.756222] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [11948.060466] Key type lgssc unregistered [11948.367881] LNet: 315135:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11948.377137] LNetError: 315135:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11948.392713] LNet: Removed LNI 192.168.204.135@tcp [11949.439181] Key type .llcrypt unregistered [11949.441610] Key type ._llcrypt unregistered [11980.803900] Key type ._llcrypt registered [11980.807074] Key type .llcrypt registered [11980.943656] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [11996.465521] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [11997.642589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11997.694787] alg: No test for adler32 (adler32-zlib) [11998.797398] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [11999.063785] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [12000.763290] Key type lgssc registered [12001.895109] Lustre: Echo OBD driver; http://www.lustre.org/ [12035.899689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12035.932104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12037.176939] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12037.212715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12037.330018] Lustre: lustre-MDT0000: new disk, initializing [12037.433991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12037.446565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12042.238179] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12052.885842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12053.020292] Lustre: 318996:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12053.089311] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12053.094942] Lustre: Skipped 1 previous similar message [12053.170491] Lustre: lustre-MDT0001: new disk, initializing [12053.309858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12053.374443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12053.391786] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12057.892603] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12067.173195] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12073.488248] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12081.460288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12081.706908] Lustre: lustre-OST0000: new disk, initializing [12081.712958] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12081.725216] Lustre: 320965:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12081.836242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12087.756592] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12088.909706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12088.919524] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12089.011777] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12097.324511] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12099.958186] Lustre: Mounted lustre-client [12101.256866] Lustre: Unmounted lustre-client [12114.914383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12114.916985] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12114.923216] LustreError: Skipped 1 previous similar message [12114.932545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12114.980949] Lustre: Skipped 1 previous similar message [12115.434094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12115.442868] Lustre: Skipped 2 previous similar messages [12120.546904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12120.559161] Lustre: Skipped 1 previous similar message [12125.675454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12125.691859] Lustre: Skipped 1 previous similar message [12126.176184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12126.420553] Lustre: server umount lustre-OST0000 complete [12130.792990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12130.808117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12135.393163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12135.408296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12136.614470] Lustre: server umount lustre-MDT0000 complete [12139.980848] LustreError: 319790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782254033 with bad export cookie 14114769106521764634 [12139.984909] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12139.989956] LustreError: 319790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12140.232847] Lustre: server umount lustre-MDT0001 complete [12152.256331] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [12154.650555] Key type lgssc unregistered [12155.744667] LNet: 1 local NIs in recovery (showing 1): 192.168.204.135@tcp [12156.000767] LNet: 323126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12156.008253] LNetError: 323126:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12231.782227] LNet: Removed LNI 192.168.204.135@tcp [12232.506537] Key type .llcrypt unregistered [12232.508896] Key type ._llcrypt unregistered [12255.684959] Key type ._llcrypt registered [12255.696411] Key type .llcrypt registered [12255.866497] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [12275.973319] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [12277.482823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12277.500652] alg: No test for adler32 (adler32-zlib) [12278.719594] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [12279.034248] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [12280.736170] Key type lgssc registered [12281.912328] Lustre: Echo OBD driver; http://www.lustre.org/ [12312.136535] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 18:36:45 (1782254205) [12318.436748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12318.455543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12319.706989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12319.733978] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12319.837818] Lustre: lustre-MDT0000: new disk, initializing [12319.928665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12319.952721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12324.396653] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12335.352915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12335.476326] Lustre: 326348:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12335.538098] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12335.541078] Lustre: Skipped 1 previous similar message [12335.617149] Lustre: lustre-MDT0001: new disk, initializing [12335.732677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12335.778646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12335.799378] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12340.598591] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12351.053964] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12356.984880] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12363.633058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12363.936354] Lustre: lustre-OST0000: new disk, initializing [12363.940328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12363.946367] Lustre: 328319:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12364.010592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12370.003800] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12372.047932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12372.060107] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12372.107718] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12381.244579] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12403.173213] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12403.179805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12403.183093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12403.186847] Lustre: Skipped 1 previous similar message [12403.199780] Lustre: Skipped 1 previous similar message [12405.434858] Lustre: server umount lustre-OST0000 complete [12412.389798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12412.392954] LustreError: Skipped 1 previous similar message [12412.395676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12412.406627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12413.412685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12413.413680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12413.453877] Lustre: Skipped 1 previous similar message [12416.229422] Lustre: server umount lustre-MDT0000 complete [12419.846961] LustreError: 326339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782254313 with bad export cookie 16408415178828093883 [12419.859872] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12419.863586] LustreError: 326339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12420.325611] Lustre: server umount lustre-MDT0001 complete [12433.437542] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [12436.153202] Key type lgssc unregistered [12436.375928] LNet: 330226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12436.383937] LNetError: 330226:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12436.410968] LNet: Removed LNI 192.168.204.135@tcp [12437.196446] Key type .llcrypt unregistered [12437.198693] Key type ._llcrypt unregistered [12458.251643] Key type ._llcrypt registered [12458.253124] Key type .llcrypt registered [12458.369620] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [12459.416411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12459.481073] alg: No test for adler32 (adler32-zlib) [12460.621341] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [12460.885332] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [12462.576189] Key type lgssc registered [12463.751044] Lustre: Echo OBD driver; http://www.lustre.org/ [12471.183523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12471.196981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12472.653799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12477.121423] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12482.018938] LustreError: 331502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12482.036552] LustreError: 331502:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12485.258261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12485.686430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12490.306978] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12498.394792] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12504.884405] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12513.827548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12514.343404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12521.796721] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12530.780535] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12541.619137] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12543.183263] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12548.144780] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.35@tcp (stopping) [12551.142598] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12551.144781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12551.165371] Lustre: Skipped 2 previous similar messages [12551.188443] Lustre: Skipped 2 previous similar messages [12553.252619] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.35@tcp (stopping) [12553.754444] Lustre: server umount lustre-MDT0000 complete [12556.259568] LustreError: 332503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12556.292073] LustreError: 332503:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12558.346479] LustreError: 331498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12558.535280] LustreError: 331484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782254452 with bad export cookie 496957136500254593 [12558.543094] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12558.556370] LustreError: 331484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12561.380064] LustreError: 331499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12561.381802] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12561.426078] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12564.975741] Lustre: server umount lustre-MDT0001 complete [12569.104036] Lustre: server umount lustre-OST0000 complete [12578.204069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12583.889953] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12586.963491] Lustre: 335682:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.135@tcp [12589.338296] Lustre: server umount MGS complete [12599.263471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12599.777500] LustreError: 336255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12599.798406] LustreError: 336255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12599.866419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12605.291073] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12608.336931] LustreError: 336252:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12608.362499] LustreError: 336252:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12615.930926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12616.335663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12621.808803] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12632.351618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12632.768980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12640.680295] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12648.926655] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12650.840356] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12656.545604] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12658.224568] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12664.811512] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12664.823261] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12664.835158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12664.839827] Lustre: Skipped 2 previous similar messages [12665.603814] Lustre: server umount lustre-OST0000 complete [12670.948269] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12670.962145] Lustre: Skipped 1 previous similar message [12675.378316] Lustre: server umount lustre-MDT0000 complete [12676.069155] LustreError: 336255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12676.090624] LustreError: 336255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12679.382663] LustreError: 336237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782254573 with bad export cookie 496957136500255433 [12679.397821] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12690.009914] Lustre: server umount lustre-MDT0001 complete [12700.653186] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [12703.734900] Key type lgssc unregistered [12704.048640] LNet: 339692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12704.059132] LNetError: 339692:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12704.089078] LNet: Removed LNI 192.168.204.135@tcp [12704.868309] Key type .llcrypt unregistered [12704.872587] Key type ._llcrypt unregistered [12724.617259] Key type ._llcrypt registered [12724.620231] Key type .llcrypt registered [12724.732171] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [12741.231467] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [12742.674992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12742.746952] alg: No test for adler32 (adler32-zlib) [12743.844657] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [12744.141459] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [12745.872337] Key type lgssc registered [12747.180437] Lustre: Echo OBD driver; http://www.lustre.org/ [12784.279707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12784.307221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12785.550480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12785.579630] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12785.664612] Lustre: lustre-MDT0000: new disk, initializing [12785.786989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12785.816057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12790.462602] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12801.456218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12801.607213] Lustre: 342855:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12801.675817] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12801.679953] Lustre: Skipped 1 previous similar message [12801.766411] Lustre: lustre-MDT0001: new disk, initializing [12801.944885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12802.002737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12802.021746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12806.678909] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12815.371801] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12820.896746] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12828.125202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12828.323959] Lustre: lustre-OST0000: new disk, initializing [12828.326802] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12828.336523] Lustre: 344826:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12828.456670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12835.601284] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12836.446862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12836.458357] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12836.595283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12847.670894] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12853.003626] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12853.243145] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12858.804236] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12859.048407] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12862.444819] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12862.445580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12862.447976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12862.460498] LustreError: Skipped 1 previous similar message [12862.514381] Lustre: Skipped 1 previous similar message [12867.565575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12867.572492] Lustre: Skipped 1 previous similar message [12872.688817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12872.698977] Lustre: Skipped 1 previous similar message [12875.232442] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12875.488967] Lustre: server umount lustre-OST0000 complete [12882.913642] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12882.920031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12882.935456] Lustre: Skipped 2 previous similar messages [12882.959688] Lustre: Skipped 1 previous similar message [12885.480682] Lustre: server umount lustre-MDT0000 complete [12890.045777] LustreError: 343651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782254784 with bad export cookie 10215899073367079218 [12890.062250] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12890.066433] LustreError: 343651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12890.493808] Lustre: server umount lustre-MDT0001 complete [12900.100976] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 18:46:33 (1782254793) [12918.093131] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [12928.669807] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [12967.095805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12967.275989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12967.305303] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12967.388173] Lustre: lustre-MDT0000: new disk, initializing [12967.474393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12967.491711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12972.209735] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12983.057189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12983.170840] Lustre: 349590:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12983.249231] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12983.263450] Lustre: Skipped 1 previous similar message [12983.373566] Lustre: lustre-MDT0001: new disk, initializing [12983.510341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12983.561137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12983.589255] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12988.673775] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12998.328854] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13003.943026] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13011.365696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13011.818941] Lustre: lustre-OST0000: new disk, initializing [13011.831556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13011.844135] Lustre: 351563:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13012.012556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13018.271831] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [13020.275531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13020.299134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13020.417987] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13028.766960] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13066.724903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13066.732685] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13066.735033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13066.735169] Lustre: Skipped 1 previous similar message [13066.740294] LustreError: Skipped 1 previous similar message [13069.793531] Lustre: server umount lustre-OST0000 complete [13075.937400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13075.955621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13075.989595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13075.999065] Lustre: Skipped 1 previous similar message [13080.446742] Lustre: server umount lustre-MDT0000 complete [13082.082507] LustreError: 350392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13082.120973] LustreError: 350392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13084.575957] LustreError: 349582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782254978 with bad export cookie 10215899073367080107 [13084.577569] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13084.590184] LustreError: 349582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13085.142754] Lustre: server umount lustre-MDT0001 complete [13095.592207] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [13098.642232] Key type lgssc unregistered [13099.016995] LNet: 353737:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13099.029481] LNetError: 353737:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13099.044221] LNet: Removed LNI 192.168.204.135@tcp [13099.972354] Key type .llcrypt unregistered [13099.975487] Key type ._llcrypt unregistered [13192.315712] Key type ._llcrypt registered [13192.319872] Key type .llcrypt registered [13192.437915] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [13192.792503] Key type .llcrypt unregistered [13192.802173] Key type ._llcrypt unregistered [13277.117662] Key type ._llcrypt registered [13277.120766] Key type .llcrypt registered [13277.309476] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [13277.790393] Key type .llcrypt unregistered [13277.795564] Key type ._llcrypt unregistered [13311.693882] Key type ._llcrypt registered [13311.697811] Key type .llcrypt registered [13311.828174] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [13312.155545] Key type .llcrypt unregistered [13312.159875] Key type ._llcrypt unregistered [13340.956369] Key type ._llcrypt registered [13340.962395] Key type .llcrypt registered [13341.127745] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [13341.503430] Key type .llcrypt unregistered [13341.510320] Key type ._llcrypt unregistered [13416.389655] Key type ._llcrypt registered [13416.392717] Key type .llcrypt registered [13416.550838] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [13416.928879] Key type .llcrypt unregistered [13416.941766] Key type ._llcrypt unregistered [13448.823682] Key type ._llcrypt registered [13448.829799] Key type .llcrypt registered [13448.964831] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [13449.260252] Key type .llcrypt unregistered [13449.262318] Key type ._llcrypt unregistered [13480.301948] Key type ._llcrypt registered [13480.304237] Key type .llcrypt registered [13480.391290] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [13480.701235] Key type .llcrypt unregistered [13480.704547] Key type ._llcrypt unregistered [13512.494755] Key type ._llcrypt registered [13512.496536] Key type .llcrypt registered [13512.600375] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [13512.880616] Key type .llcrypt unregistered [13512.883306] Key type ._llcrypt unregistered [13540.526749] Key type ._llcrypt registered [13540.531186] Key type .llcrypt registered [13540.703686] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [13540.888227] Key type .llcrypt unregistered [13540.891632] Key type ._llcrypt unregistered [13577.809366] Key type ._llcrypt registered [13577.811064] Key type .llcrypt registered [13578.008294] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [13578.394456] Key type .llcrypt unregistered [13578.396401] Key type ._llcrypt unregistered [13642.426195] Key type ._llcrypt registered [13642.428321] Key type .llcrypt registered [13642.531176] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [13642.786502] Key type .llcrypt unregistered [13642.790832] Key type ._llcrypt unregistered [13671.301652] Key type ._llcrypt registered [13671.305525] Key type .llcrypt registered [13671.466837] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [13671.820400] Key type .llcrypt unregistered [13671.823792] Key type ._llcrypt unregistered [13704.650955] Key type ._llcrypt registered [13704.653538] Key type .llcrypt registered [13704.840383] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [13705.219957] Key type .llcrypt unregistered [13705.221874] Key type ._llcrypt unregistered [13759.832582] Key type ._llcrypt registered [13759.834786] Key type .llcrypt registered [13759.942438] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [13774.868705] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [13776.257792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13776.366631] alg: No test for adler32 (adler32-zlib) [13777.434759] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [13777.602198] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [13779.273062] Key type lgssc registered [13780.635800] Lustre: Echo OBD driver; http://www.lustre.org/ [13819.865558] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13819.904440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13821.154585] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13821.174239] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13821.301471] Lustre: tst250fs-MDT0000: new disk, initializing [13821.410939] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13821.433977] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13825.956449] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [13838.008901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13838.173483] Lustre: 374068:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13838.264127] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13838.269109] Lustre: Skipped 1 previous similar message [13838.404711] Lustre: tst250fs-MDT0001: new disk, initializing [13838.534810] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13838.571179] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13838.595373] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13845.058643] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [13850.412370] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13858.950593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13859.558862] Lustre: tst250fs-OST0000: new disk, initializing [13859.562863] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13859.569580] Lustre: 375698:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13859.745880] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13861.136785] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13861.160473] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13861.339381] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13867.372581] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [13880.550395] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13880.682063] Lustre: tst250fs-OST0001: new disk, initializing [13880.685657] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13880.693352] Lustre: 376574:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13880.752522] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13886.785080] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [13888.053904] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13888.067480] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13888.150414] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13891.900573] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13894.653316] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13894.659755] Lustre: Skipped 2 previous similar messages [13914.084872] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13914.087509] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13914.117603] Lustre: Skipped 3 previous similar messages [13914.124428] Lustre: Skipped 2 previous similar messages [13918.688511] Lustre: server umount tst250fs-MDT0000 complete [13922.498975] LustreError: 374059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782255816 with bad export cookie 8962248680564719885 [13922.510467] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13922.523490] LustreError: 374059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13922.892272] Lustre: server umount tst250fs-MDT0001 complete [13936.323760] Lustre: server umount tst250fs-OST0000 complete [13940.197613] Lustre: 372289:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782255818/real 1782255818] req@ffff9a78432eb480 x1868830522546688/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782255834 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13940.236968] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13940.526732] Lustre: server umount tst250fs-OST0001 complete [13959.264813] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [13967.538936] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [14001.887250] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 19:04:54 (1782255894) [14003.595682] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14025.937859] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [14037.120696] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [14073.266837] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13822 sec ========= 19:06:05 (1782255965) [14075.410550] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:06:08 (1782255968) === [14079.879525] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:06:12 (1782255972) === [14125.479857] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [14128.553372] Key type lgssc unregistered [14128.944248] LNet: 385722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14128.957178] LNetError: 385722:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14128.982028] LNet: Removed LNI 192.168.204.135@tcp [14129.952321] Key type .llcrypt unregistered [14129.958675] Key type ._llcrypt unregistered