[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 486732062 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003095] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009024] pid_max: default: 32768 minimum: 301 [ 0.010136] LSM: Security Framework initializing [ 0.012021] Yama: becoming mindful. [ 0.012951] SELinux: Initializing. [ 0.013068] *** VALIDATE selinux *** [ 0.021582] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026829] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027168] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029109] *** VALIDATE tmpfs *** [ 0.031266] *** VALIDATE proc *** [ 0.032258] *** VALIDATE cgroup *** [ 0.033014] *** VALIDATE cgroup2 *** [ 0.034275] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035169] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037031] Spectre V2 : User space: Vulnerable [ 0.038011] Speculative Store Bypass: Vulnerable [ 0.041249] debug: unmapping init [mem 0xffffffff8b859000-0xffffffff8b860fff] [ 0.044000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044745] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045024] ... version: 2 [ 0.046014] ... bit width: 48 [ 0.047015] ... generic registers: 4 [ 0.048011] ... value mask: 0000ffffffffffff [ 0.049014] ... max period: 00007fffffffffff [ 0.050019] ... fixed-purpose events: 3 [ 0.051013] ... event mask: 000000070000000f [ 0.052331] rcu: Hierarchical SRCU implementation. [ 0.054496] smp: Bringing up secondary CPUs ... [ 0.055687] x86: Booting SMP configuration: [ 0.056029] .... node #0, CPUs: #1 #2 #3 [ 0.059374] smp: Brought up 1 node, 4 CPUs [ 0.061015] smpboot: Max logical packages: 1 [ 0.062022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.156036] node 0 deferred pages initialised in 91ms [ 0.159265] devtmpfs: initialized [ 0.160226] x86/mm: Memory block size: 128MB [ 0.163730] gcov: version magic: 0x41383552 [ 0.166329] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.167097] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.168357] pinctrl core: initialized pinctrl subsystem [ 0.170037] [ 0.170630] ************************************************************* [ 0.173015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175015] ** ** [ 0.177014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.179016] ** ** [ 0.181014] ** This means that this kernel is built to expose internal ** [ 0.184017] ** IOMMU data structures, which may compromise security on ** [ 0.185012] ** your system. ** [ 0.187013] ** ** [ 0.189013] ** If you see this message and you are not debugging the ** [ 0.191018] ** kernel, report this immediately to your vendor! ** [ 0.193013] ** ** [ 0.195016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.198026] ************************************************************* [ 0.200731] NET: Registered protocol family 16 [ 0.202449] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.205067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.208105] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.212020] cpuidle: using governor menu [ 0.213592] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.214475] PCI: Using configuration type 1 for base access [ 0.215130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.223070] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.225031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.229058] cryptd: max_cpu_qlen set to 1000 [ 0.232000] ACPI: Added _OSI(Module Device) [ 0.233017] ACPI: Added _OSI(Processor Device) [ 0.234014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.235016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.240187] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.248714] ACPI: Interpreter enabled [ 0.250078] ACPI: PM: (supports S0 S3 S4 S5) [ 0.252018] ACPI: Using IOAPIC for interrupt routing [ 0.254134] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.256456] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.266565] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.269052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.271023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.274112] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.279490] acpiphp: Slot [2] registered [ 0.281164] acpiphp: Slot [5] registered [ 0.282226] acpiphp: Slot [6] registered [ 0.284133] acpiphp: Slot [7] registered [ 0.285121] acpiphp: Slot [8] registered [ 0.286190] acpiphp: Slot [9] registered [ 0.287131] acpiphp: Slot [10] registered [ 0.289132] acpiphp: Slot [3] registered [ 0.290088] acpiphp: Slot [4] registered [ 0.291216] acpiphp: Slot [11] registered [ 0.293146] acpiphp: Slot [12] registered [ 0.295145] acpiphp: Slot [13] registered [ 0.296229] acpiphp: Slot [14] registered [ 0.298124] acpiphp: Slot [15] registered [ 0.299116] acpiphp: Slot [16] registered [ 0.301162] acpiphp: Slot [17] registered [ 0.302119] acpiphp: Slot [18] registered [ 0.304135] acpiphp: Slot [19] registered [ 0.306123] acpiphp: Slot [20] registered [ 0.307131] acpiphp: Slot [21] registered [ 0.309134] acpiphp: Slot [22] registered [ 0.310148] acpiphp: Slot [23] registered [ 0.311134] acpiphp: Slot [24] registered [ 0.313129] acpiphp: Slot [25] registered [ 0.314103] acpiphp: Slot [26] registered [ 0.315122] acpiphp: Slot [27] registered [ 0.317118] acpiphp: Slot [28] registered [ 0.318105] acpiphp: Slot [29] registered [ 0.320147] acpiphp: Slot [30] registered [ 0.321132] acpiphp: Slot [31] registered [ 0.322118] PCI host bridge to bus 0000:00 [ 0.324030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.326040] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.328027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.330028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.333041] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.335039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.337207] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.340013] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.343384] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.353019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.357527] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.358023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.360021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.363030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.365556] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.367753] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.370048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.372919] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.378021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.391025] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.396031] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.402758] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.410024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.416026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.438016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.446689] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.454000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.461016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.476017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.484000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.497015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.505020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.537020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.545000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.551016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.555015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.578017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.595008] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.607016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.619016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.652016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.671559] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.683019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.697017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.724017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.735846] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.737322] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.739346] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.740340] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.742213] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.748033] iommu: Default domain type: Passthrough [ 0.749399] SCSI subsystem initialized [ 0.750114] ACPI: bus type USB registered [ 0.751103] usbcore: registered new interface driver usbfs [ 0.753066] usbcore: registered new interface driver hub [ 0.754074] usbcore: registered new device driver usb [ 0.755175] pps_core: LinuxPPS API ver. 1 registered [ 0.756010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.758080] PTP clock support registered [ 0.762050] EDAC MC: Ver: 3.0.0 [ 0.764132] PCI: Using ACPI for IRQ routing [ 0.765890] NetLabel: Initializing [ 0.767012] NetLabel: domain hash size = 128 [ 0.768010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.771090] NetLabel: unlabeled traffic allowed by default [ 0.773106] vgaarb: loaded [ 0.774275] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.776017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.783000] clocksource: Switched to clocksource kvm-clock [ 0.889295] VFS: Disk quotas dquot_6.6.0 [ 0.890641] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.893144] *** VALIDATE ramfs *** [ 0.894377] *** VALIDATE hugetlbfs *** [ 0.896468] pnp: PnP ACPI init [ 0.898911] pnp: PnP ACPI: found 6 devices [ 0.913992] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.917331] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.919548] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.921771] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.924121] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.926592] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.929578] NET: Registered protocol family 2 [ 0.932119] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.937286] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.940452] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.945662] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.951094] TCP: Hash tables configured (established 65536 bind 65536) [ 0.955085] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.959448] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.963443] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.967704] NET: Registered protocol family 1 [ 0.970934] RPC: Registered named UNIX socket transport module. [ 0.974163] RPC: Registered udp transport module. [ 0.976742] RPC: Registered tcp transport module. [ 0.979300] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.983245] NET: Registered protocol family 44 [ 0.984880] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.986876] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.988965] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.991317] PCI: CLS 0 bytes, default 64 [ 0.993568] Unpacking initramfs... [ 2.424429] debug: unmapping init [mem 0xffff9fc8fcc54000-0xffff9fc8fffbffff] [ 2.428055] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.429744] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.431987] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.922592] Initialise system trusted keyrings [ 2.924254] Key type blacklist registered [ 2.927292] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.936564] zbud: loaded [ 2.939608] *** VALIDATE nfs *** [ 2.940965] *** VALIDATE nfs4 *** [ 2.942620] pstore: using deflate compression [ 2.946553] Platform Keyring initialized [ 3.067463] NET: Registered protocol family 38 [ 3.069676] Key type asymmetric registered [ 3.071316] Asymmetric key parser 'x509' registered [ 3.073360] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.076686] io scheduler mq-deadline registered [ 3.078455] io scheduler kyber registered [ 3.079974] io scheduler bfq registered [ 3.081631] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.085891] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.088833] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.093921] ACPI: Power Button [PWRF] [ 3.099708] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.105833] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.120981] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.127034] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.142296] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.172399] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.203033] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.208410] Non-volatile memory driver v1.3 [ 3.209865] Linux agpgart interface v0.103 [ 3.243522] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.246250] vda: detected capacity change from 0 to 76677120 [ 3.264232] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.266613] vdb: detected capacity change from 0 to 1073741824 [ 3.282663] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.285776] vdc: detected capacity change from 0 to 2621440000 [ 3.300856] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.303761] vdd: detected capacity change from 0 to 2621440000 [ 3.321528] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.324243] vde: detected capacity change from 0 to 4294967296 [ 3.340980] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.346325] vdf: detected capacity change from 0 to 4294967296 [ 3.353838] libphy: Fixed MDIO Bus: probed [ 3.362152] usbcore: registered new interface driver usbserial_generic [ 3.363973] usbserial: USB Serial support registered for generic [ 3.365766] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.368992] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.370702] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.373091] mousedev: PS/2 mouse device common for all mice [ 3.375581] rtc_cmos 00:05: RTC can wake from S4 [ 3.378943] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.381535] rtc_cmos 00:05: registered as rtc0 [ 3.383774] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.385983] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.387444] intel_pstate: CPU model not supported [ 3.393139] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.398105] hid: raw HID events driver (C) Jiri Kosina [ 3.399800] usbcore: registered new interface driver usbhid [ 3.401878] usbhid: USB HID core driver [ 3.403553] drop_monitor: Initializing network drop monitor service [ 3.406274] Initializing XFRM netlink socket [ 3.408338] NET: Registered protocol family 10 [ 3.411689] Segment Routing with IPv6 [ 3.413073] NET: Registered protocol family 17 [ 3.415535] mpls_gso: MPLS GSO support [ 3.420717] RAS: Correctable Errors collector initialized. [ 3.423047] AVX version of gcm_enc/dec engaged. [ 3.424690] AES CTR mode by8 optimization enabled [ 3.520397] sched_clock: Marking stable (3520364577, 0)->(4429069933, -908705356) [ 3.524271] registered taskstats version 1 [ 3.526386] Loading compiled-in X.509 certificates [ 3.529715] zswap: loaded using pool lzo/zbud [ 3.556126] Key type big_key registered [ 3.569446] Key type encrypted registered [ 3.570946] ima: No TPM chip found, activating TPM-bypass! [ 3.573127] ima: Allocated hash algorithm: sha1 [ 3.574210] ima: No architecture policies found [ 3.575366] evm: Initialising EVM extended attributes: [ 3.577315] evm: security.selinux [ 3.578599] evm: security.ima [ 3.579634] evm: security.capability [ 3.580988] evm: HMAC attrs: 0x1 [ 3.583308] rtc_cmos 00:05: setting system clock to 2026-09-03 11:56:28 UTC (1788436588) [ 3.589660] debug: unmapping init [mem 0xffffffff8c803000-0xffffffff8c9fffff] [ 3.592215] debug: unmapping init [mem 0xffffffff8b582000-0xffffffff8b858fff] [ 3.604475] Write protecting the kernel read-only data: 28672k [ 3.607697] debug: unmapping init [mem 0xffffffff89c03000-0xffffffff89dfffff] [ 3.611994] debug: unmapping init [mem 0xffffffff8a514000-0xffffffff8a5fffff] [ 3.647201] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.656457] systemd[1]: Detected virtualization kvm. [ 3.659475] systemd[1]: Detected architecture x86-64. [ 3.661290] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.692166] systemd[1]: No hostname configured. [ 3.693847] systemd[1]: Set hostname to . [ 3.696029] random: systemd: uninitialized urandom read (16 bytes read) [ 3.698628] systemd[1]: Initializing machine ID from random generator. [ 3.751659] random: ln: uninitialized urandom read (6 bytes read) [ 3.852741] random: systemd: uninitialized urandom read (16 bytes read) [ 3.855505] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.860337] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.865396] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.507231] device-mapper: uevent: version 1.0.3 [ 4.509413] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.480518] virtio_net virtio0 ens2: renamed from eth0 [ 5.545589] random: fast init done [ 5.804752] scsi host0: ata_piix [ 5.870345] scsi host1: ata_piix [ 5.874974] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.885198] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.090156] random: crng init done [ 11.091711] random: 7 urandom warning(s) missed due to ratelimiting [ 12.843733] dracut-initqueue[589]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 14.039497] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.160927] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.009327] SELinux: Disabled at runtime. [ 17.146834] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.176746] systemd[1]: Detected virtualization kvm. [ 17.178457] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.671958] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.676258] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.683983] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.688699] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.692610] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.701771] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.714938] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ 18.909730] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 19.847907] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.401463] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.513473] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.676996] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.711768] EDAC sbridge: Ver: 1.1.2 [ 25.344491] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (6s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 26.437506] NFS: Registering the id_resolver key type [ 26.442088] Key type id_resolver registered [ 26.443545] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg408-server login: [ 95.701678] libcfs: loading out-of-tree module taints kernel. [ 95.739364] Key type ._llcrypt registered [ 95.740925] Key type .llcrypt registered [ 95.839407] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 119.762217] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 123.024198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 123.098641] alg: No test for adler32 (adler32-zlib) [ 124.694841] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 126.492481] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 128.279194] Key type lgssc registered [ 129.902692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 145.685345] hrtimer: interrupt took 7336002 ns [ 148.201172] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 193.999994] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 208.539641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 208.561727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.944885] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 210.015593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 210.132610] Lustre: lustre-MDT0000: new disk, initializing [ 210.265763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 210.318995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 215.687788] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.408781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 231.551597] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 231.602933] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 231.608716] Lustre: Skipped 1 previous similar message [ 231.771876] Lustre: lustre-MDT0001: new disk, initializing [ 231.883501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 231.922080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 231.935127] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 237.078054] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.784627] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 254.986384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 255.355482] Lustre: lustre-OST0000: new disk, initializing [ 255.362536] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 255.374047] Lustre: 8455:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 255.447112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 262.472560] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 263.719379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 263.730949] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 263.836984] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 280.498331] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 280.680374] Lustre: lustre-OST0001: new disk, initializing [ 280.684421] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 280.692151] Lustre: 9543:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 280.813153] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 288.819740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 288.830205] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 288.945157] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 288.977835] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 303.809619] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 312.074978] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 319.643484] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing check_logdir /tmp/testlogs/ [ 325.293512] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing yml_node [ 329.676159] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 333.273260] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 335.860696] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 337.412080] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:02:00 EDT 2026 [ 355.934514] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 357.808689] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 375.264896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 375.278508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 375.290303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 376.289092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 376.289887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 376.293592] Lustre: Skipped 1 previous similar message [ 376.318577] Lustre: Skipped 1 previous similar message [ 381.627583] Lustre: server umount lustre-MDT0000 complete [ 386.444734] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788436971 with bad export cookie 11354972382741241503 [ 386.450738] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 386.453308] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 386.528059] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.530689] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 386.544936] Lustre: Skipped 1 previous similar message [ 386.545823] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 386.545828] Lustre: Skipped 2 previous similar messages [ 386.579420] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 391.647950] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 391.648586] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 391.683489] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 391.702610] Lustre: Skipped 2 previous similar messages [ 393.146815] Lustre: server umount lustre-MDT0001 complete [ 399.232612] Lustre: server umount lustre-OST0000 complete [ 405.057063] Lustre: server umount lustre-OST0001 complete [ 413.099548] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 423.480356] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 458.874052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 459.072575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 459.091818] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 459.150971] Lustre: lustre-MDT0000: new disk, initializing [ 459.228113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 459.257153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 463.606979] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 474.204252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 474.386130] Lustre: 15097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 474.438382] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 474.442658] Lustre: Skipped 1 previous similar message [ 474.507799] Lustre: lustre-MDT0001: new disk, initializing [ 474.614916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 474.649440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 474.664529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 479.277539] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 489.782395] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 496.136570] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 504.465650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 504.728329] Lustre: lustre-OST0000: new disk, initializing [ 504.732370] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 504.740704] Lustre: 17064:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 504.849406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 505.955868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 505.989668] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 506.081029] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 510.671671] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 521.190250] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 528.334650] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 528.758466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 536.379669] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 536.800213] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 542.177792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 542.187594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 542.188862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 542.202819] Lustre: Skipped 2 previous similar messages [ 542.228888] Lustre: Skipped 1 previous similar message [ 545.850556] Lustre: server umount lustre-OST0000 complete [ 551.393113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 551.404787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 551.435639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 556.456708] Lustre: server umount lustre-MDT0000 complete [ 557.552270] LustreError: 15107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.572013] LustreError: 15107:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 560.486343] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437145 with bad export cookie 11354972382741243036 [ 560.510677] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 560.511572] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 560.917632] Lustre: server umount lustre-MDT0001 complete [ 567.917303] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 08:05:51 (1788437151) [ 587.338592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 587.790932] LustreError: 19939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 587.865314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 592.064354] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 592.873973] LustreError: 19940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 600.679367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 605.772611] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 614.635936] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 620.346293] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 622.363105] Lustre: Failing over lustre-MDT0000 [ 622.564930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 622.575987] Lustre: Skipped 1 previous similar message [ 622.595808] LustreError: 20667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 622.611013] LustreError: 20667:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 622.748914] Lustre: server umount lustre-MDT0000 complete [ 641.007704] LustreError: 19939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 641.029988] LustreError: 19939:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 642.015670] Lustre: 3649:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788437210/real 1788437210] req@ffff9fc8436c1180 x1875311812840704/t0(0) o400->MGC192.168.204.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788437226 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 642.068146] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 642.153583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 653.288820] LustreError: 3647:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fc843688e00 x1875311812843648/t0(0) o250->MGC192.168.204.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 653.691830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 653.697864] Lustre: Skipped 1 previous similar message [ 653.754119] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 657.773943] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 658.919525] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 658.930084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 658.974378] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 668.107278] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 674.276451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 674.293400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 674.314696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 674.327849] Lustre: Skipped 2 previous similar messages [ 679.426412] Lustre: server umount lustre-MDT0000 complete [ 680.415734] LustreError: 19939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 680.440596] LustreError: 19939:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 683.462597] LustreError: 19921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437268 with bad export cookie 11354972382741244450 [ 683.474367] LustreError: 19921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 683.478712] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 683.800606] Lustre: server umount lustre-MDT0001 complete [ 693.066408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 699.076369] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 707.305039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.578749] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 720.220246] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 726.486861] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 728.516881] Lustre: Failing over lustre-MDT0000 [ 729.058266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 729.073985] Lustre: Skipped 2 previous similar messages [ 730.698677] Lustre: server umount lustre-MDT0000 complete [ 744.224366] Lustre: 3648:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788437313/real 1788437313] req@ffff9fc96bd47480 x1875311812870400/t0(0) o400->MGC192.168.204.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788437329 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 744.270522] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 749.538255] LustreError: 23279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 749.560868] LustreError: 23279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 750.275323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 754.664101] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x9d94f7388ba9adcc [ 754.693283] LustreError: 25283:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.204.108@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (ffff9fc843012700) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 754.730233] Lustre: MGC192.168.204.108@tcp: Connection restored to 0@lo (at 0@lo) [ 754.738240] Lustre: Skipped 1 previous similar message [ 755.060390] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 759.999818] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 760.289976] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 760.304456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 760.333148] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 770.775711] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 775.648547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 775.658304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 775.662421] Lustre: Skipped 2 previous similar messages [ 781.166577] Lustre: server umount lustre-MDT0000 complete [ 784.868559] LustreError: 23263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437369 with bad export cookie 11354972382741245388 [ 784.885292] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 785.268338] Lustre: server umount lustre-MDT0001 complete [ 794.266604] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 08:09:37 (1788437377) [ 812.554318] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 823.214383] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 865.375130] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 876.798313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 877.117528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 877.195841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 877.281342] Lustre: lustre-MDT0000: new disk, initializing [ 877.402881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 877.408764] Lustre: Skipped 3 previous similar messages [ 877.431214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 882.105706] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 893.776154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 893.973899] Lustre: 29656:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 894.332487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 898.898899] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 908.949456] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 914.624423] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 921.606662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 921.929700] Lustre: 31626:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 923.808538] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 923.838431] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 923.916721] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 923.933220] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 928.650175] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 938.667229] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 945.269329] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 970.213726] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 970.227373] Lustre: Skipped 2 previous similar messages [ 970.231807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 970.234625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 970.242620] Lustre: Skipped 2 previous similar messages [ 972.486855] Lustre: server umount lustre-OST0000 complete [ 990.175154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 990.385844] Lustre: server umount lustre-MDT0000 complete [ 990.694136] LustreError: 29666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 990.708453] LustreError: 29666:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 994.909695] LustreError: 30449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437579 with bad export cookie 11354972382741245801 [ 994.912695] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 994.917747] LustreError: 30449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 995.304807] Lustre: server umount lustre-MDT0001 complete [ 1008.782367] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 1012.299703] Key type lgssc unregistered [ 1012.671152] LNet: 33606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1012.682361] LNetError: 33606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.733740] LNet: Removed LNI 192.168.204.108@tcp [ 1014.878143] Key type .llcrypt unregistered [ 1014.881187] Key type ._llcrypt unregistered [ 1040.336221] Key type ._llcrypt registered [ 1040.339823] Key type .llcrypt registered [ 1040.438858] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 1041.425584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1041.470686] alg: No test for adler32 (adler32-zlib) [ 1042.594696] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1042.820734] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 1044.559179] Key type lgssc registered [ 1045.665328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1053.653808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1053.678658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1055.212889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1060.217969] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1065.447385] LustreError: 35017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1065.463937] LustreError: 35017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1069.104810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.508267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1074.452917] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1081.452356] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1086.909218] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1094.373798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1094.658526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1100.797197] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1100.897273] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1109.977861] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1114.159680] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1127.663810] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 08:15:10 (1788437710) [ 1129.447142] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1131.188436] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1138.027438] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 08:15:21 (1788437721) [ 1147.436843] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 08:15:30 (1788437730) [ 1148.526320] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1157.599453] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1157.608769] Lustre: Skipped 2 previous similar messages [ 1159.498557] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 08:15:42 (1788437742) [ 1161.713658] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1161.720655] Lustre: Skipped 2 previous similar messages [ 1170.115099] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1170.121777] Lustre: Skipped 4 previous similar messages [ 1184.944114] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 08:16:07 (1788437767) [ 1187.733736] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1191.271628] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1199.633669] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 08:16:22 (1788437782) [ 1201.957441] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1201.969605] Lustre: Skipped 6 previous similar messages [ 1215.016440] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 08:16:38 (1788437798) [ 1231.850674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1231.859561] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1231.881073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1231.896239] Lustre: Skipped 1 previous similar message [ 1235.773725] Lustre: server umount lustre-OST0000 complete [ 1242.080309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1242.090814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1242.092856] Lustre: Skipped 1 previous similar message [ 1242.096796] Lustre: Skipped 1 previous similar message [ 1245.586996] Lustre: server umount lustre-MDT0000 complete [ 1247.201318] LustreError: 35016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1247.229328] LustreError: 35016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1249.108774] LustreError: 35750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437834 with bad export cookie 1124956519693909429 [ 1249.115352] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1249.124161] LustreError: 35750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1249.407983] Lustre: server umount lustre-MDT0001 complete [ 1259.710633] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 1262.175479] Key type lgssc unregistered [ 1262.428255] LNet: 42871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1262.443520] LNetError: 42871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1263.461797] LNet: Removed LNI 192.168.204.108@tcp [ 1264.313581] Key type .llcrypt unregistered [ 1264.316870] Key type ._llcrypt unregistered [ 1280.439538] Key type ._llcrypt registered [ 1280.442192] Key type .llcrypt registered [ 1280.546479] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 1294.542838] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 1295.512661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1295.544970] alg: No test for adler32 (adler32-zlib) [ 1296.656819] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1296.876369] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 1298.527330] Key type lgssc registered [ 1299.790567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1348.109115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1348.137963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1349.400799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1349.456404] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1349.649612] Lustre: lustre-MDT0000: new disk, initializing [ 1349.803693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1349.833289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1355.993363] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1368.668417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1368.800469] Lustre: 46027:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1368.897634] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1368.902331] Lustre: Skipped 1 previous similar message [ 1368.987950] Lustre: lustre-MDT0001: new disk, initializing [ 1369.094294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1369.137604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1369.156087] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1373.392432] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1382.145700] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1387.987230] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1394.786221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1395.011382] Lustre: lustre-OST0000: new disk, initializing [ 1395.014556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1395.020786] Lustre: 47996:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1395.113177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1401.284888] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1404.997171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1405.014832] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1405.119113] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1409.938548] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1414.214969] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1414.466320] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1418.864742] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1419.030312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1425.888826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1425.890901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1425.905478] LustreError: Skipped 1 previous similar message [ 1425.906915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1425.924995] Lustre: Skipped 1 previous similar message [ 1425.946142] Lustre: Skipped 1 previous similar message [ 1427.317903] Lustre: server umount lustre-OST0000 complete [ 1435.617244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1435.629071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1435.644044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1437.835683] Lustre: server umount lustre-MDT0000 complete [ 1441.724895] LustreError: 46018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438026 with bad export cookie 3737786589019765723 [ 1441.736863] LustreError: 46018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1441.738530] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1442.089518] Lustre: server umount lustre-MDT0001 complete [ 1454.494418] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 08:20:37 (1788438037) [ 1463.488511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1464.268932] LustreError: 50456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1464.289501] LustreError: 50456:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1464.377228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1469.428545] LustreError: 50457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1470.626507] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1474.548882] LustreError: 50456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1479.652375] LustreError: 50457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1480.074490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1480.495095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1485.755455] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1493.972853] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1500.118564] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1507.033585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1507.390370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1514.044563] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1522.119045] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1531.052384] LustreError: 52985:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1531.882842] Lustre: Modifying parameter general.timeout=1 in log params [ 1533.568291] Lustre: Modifying parameter general.timeout=3 in log params [ 1533.571834] Lustre: Skipped 1 previous similar message [ 1536.091860] Lustre: Modifying parameter general.timeout=6 in log params [ 1536.095889] Lustre: Skipped 2 previous similar messages [ 1540.347837] Lustre: Modifying parameter general.timeout=11 in log params [ 1540.353000] Lustre: Skipped 4 previous similar messages [ 1548.959723] Lustre: Modifying parameter general.timeout=22 in log params [ 1548.963675] Lustre: Skipped 10 previous similar messages [ 1565.810601] Lustre: Modifying parameter general.timeout=42 in log params [ 1565.816186] Lustre: Skipped 19 previous similar messages [ 1598.413308] Lustre: Modifying parameter general.timeout=78 in log params [ 1598.415960] Lustre: Skipped 35 previous similar messages [ 1657.568705] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 08:24:00 (1788438240) [ 1660.675960] Lustre: 59930:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1662.563784] Lustre: 60028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1663.341553] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 1663.347536] Lustre: Skipped 55 previous similar messages [ 1665.939712] Lustre: 60245:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1665.951306] Lustre: 60245:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1695.129143] Lustre: 61795:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1695.141320] Lustre: 61795:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1695.175697] LustreError: 61795:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1699.187178] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 08:24:42 (1788438282) [ 1711.074466] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1711.076948] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1711.081779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1711.081787] Lustre: Skipped 2 previous similar messages [ 1711.132575] Lustre: Skipped 3 previous similar messages [ 1715.712111] Lustre: server umount lustre-MDT0000 complete [ 1716.199803] LustreError: 52162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1716.231864] LustreError: 52162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1719.810960] LustreError: 50437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438304 with bad export cookie 3737786589019766598 [ 1719.814795] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1719.818599] LustreError: 50437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1720.254332] Lustre: server umount lustre-MDT0001 complete [ 1735.222141] Lustre: server umount lustre-OST0000 complete [ 1764.164791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1764.402965] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1764.708204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1769.329663] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1779.120830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1779.208982] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1779.248860] Lustre: 64325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1784.365406] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1792.430713] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1798.760490] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1806.582830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1806.768236] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1807.102915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1807.109832] Lustre: Skipped 1 previous similar message [ 1813.724904] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1822.891858] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1825.103478] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1825.116885] Lustre: Skipped 19 previous similar messages [ 1825.171687] Lustre: 66894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1835.493329] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1835.507833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1835.533847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1835.545310] Lustre: Skipped 2 previous similar messages [ 1838.580505] Lustre: server umount lustre-OST0000 complete [ 1846.244506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1846.256529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1846.266613] Lustre: Skipped 1 previous similar message [ 1846.268090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1846.268095] Lustre: Skipped 2 previous similar messages [ 1846.290570] LustreError: Skipped 1 previous similar message [ 1848.864484] Lustre: server umount lustre-MDT0000 complete [ 1851.364619] LustreError: 64338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1851.381809] LustreError: 64338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1852.874987] LustreError: 64316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438437 with bad export cookie 3737786589019769657 [ 1852.877173] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1852.887102] LustreError: 64316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1853.374227] Lustre: server umount lustre-MDT0001 complete [ 1866.179308] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 1869.022299] Key type lgssc unregistered [ 1869.320691] LNet: 68099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1869.325858] LNetError: 68099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.337952] LNet: Removed LNI 192.168.204.108@tcp [ 1870.249404] Key type .llcrypt unregistered [ 1870.252462] Key type ._llcrypt unregistered [ 1896.839241] Key type ._llcrypt registered [ 1896.843647] Key type .llcrypt registered [ 1896.936376] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 1897.849214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1897.876144] alg: No test for adler32 (adler32-zlib) [ 1899.162545] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1899.404569] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 1901.087185] Key type lgssc registered [ 1902.295722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1911.643353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1911.660124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1913.292669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1917.939735] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1923.557480] LustreError: 69511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1923.578152] LustreError: 69511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1928.675728] LustreError: 69512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1928.713470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1929.566888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1934.857406] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1943.794413] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1951.529377] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1960.471915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1961.251280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1968.594894] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 1976.893645] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1989.982673] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1999.339163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1999.351361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1999.359441] Lustre: Skipped 1 previous similar message [ 1999.375628] Lustre: Skipped 2 previous similar messages [ 2004.370831] Lustre: server umount lustre-MDT0000 complete [ 2004.451130] LustreError: 69507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2004.466157] LustreError: 69507:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2007.561722] LustreError: 69492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438592 with bad export cookie 8845335513777120862 [ 2007.574784] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2007.857695] Lustre: server umount lustre-MDT0001 complete [ 2021.528660] Lustre: server umount lustre-OST0000 complete [ 2045.451857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2045.698593] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2045.736553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2046.001861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2050.602050] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2058.774828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2058.887601] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2058.922602] Lustre: 74386:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2059.582668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2064.485677] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2071.386363] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2076.733599] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2083.681844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2083.840169] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2084.156336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2089.921407] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2098.480603] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2100.225279] Lustre: 76961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2102.061503] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2110.944693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2110.951321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2110.963233] LustreError: Skipped 1 previous similar message [ 2110.970099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2110.984276] Lustre: Skipped 2 previous similar messages [ 2111.001869] Lustre: Skipped 1 previous similar message [ 2112.839451] Lustre: server umount lustre-OST0000 complete [ 2120.672583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2120.680688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2120.690417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2122.561848] Lustre: server umount lustre-MDT0000 complete [ 2125.714851] LustreError: 76253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438710 with bad export cookie 8845335513777121709 [ 2125.720870] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2125.724980] LustreError: 76253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2125.976344] Lustre: server umount lustre-MDT0001 complete [ 2135.222785] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 2137.617603] Key type lgssc unregistered [ 2137.905146] LNet: 78169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2137.914321] LNetError: 78169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2137.938340] LNet: Removed LNI 192.168.204.108@tcp [ 2138.879767] Key type .llcrypt unregistered [ 2138.886689] Key type ._llcrypt unregistered [ 2164.259343] Key type ._llcrypt registered [ 2164.260805] Key type .llcrypt registered [ 2164.329289] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 2164.961806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2165.078101] alg: No test for adler32 (adler32-zlib) [ 2166.098939] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2166.271076] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 2167.951511] Key type lgssc registered [ 2169.326443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2176.148676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2176.164325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2177.682590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2182.159623] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2187.234748] LustreError: 79576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2187.259816] LustreError: 79576:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2191.388887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2192.054834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2196.252780] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2203.928652] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2209.202432] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2215.934917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2216.419598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2222.247760] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2229.238363] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2234.870448] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2235.792734] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2237.575850] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2237.583943] Lustre: Skipped 1 previous similar message [ 2240.300340] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2240.302994] Lustre: Skipped 2 previous similar messages [ 2244.796578] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2244.801723] Lustre: Skipped 4 previous similar messages [ 2253.061153] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2253.067928] Lustre: Skipped 9 previous similar messages [ 2269.968917] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2269.975734] Lustre: Skipped 16 previous similar messages [ 2302.040126] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2302.046146] Lustre: Skipped 31 previous similar messages [ 2341.101414] Lustre: 87693:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2343.692916] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 08:35:26 (1788438926) [ 2345.249959] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2347.656503] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 08:35:30 (1788438930) [ 2355.684256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2355.688359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2355.691074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2355.707459] Lustre: Skipped 1 previous similar message [ 2355.710467] LustreError: Skipped 1 previous similar message [ 2358.309642] Lustre: server umount lustre-OST0000 complete [ 2365.923521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2365.933597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2365.942272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2365.943960] Lustre: Skipped 1 previous similar message [ 2365.964525] Lustre: Skipped 2 previous similar messages [ 2368.253198] Lustre: server umount lustre-MDT0000 complete [ 2371.045517] LustreError: 80307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2371.067566] LustreError: 80307:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2371.866484] LustreError: 79557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438956 with bad export cookie 15377425117448109142 [ 2371.875929] LustreError: 79557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2371.880229] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2372.361480] Lustre: server umount lustre-MDT0001 complete [ 2385.534771] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 2388.478836] Key type lgssc unregistered [ 2388.813652] LNet: 88941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2388.824537] LNetError: 88941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2388.851231] LNet: Removed LNI 192.168.204.108@tcp [ 2389.750565] Key type .llcrypt unregistered [ 2389.755174] Key type ._llcrypt unregistered [ 2394.619725] Key type ._llcrypt registered [ 2394.622979] Key type .llcrypt registered [ 2394.742778] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2399.950113] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules [ 2401.041397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2401.240097] alg: No test for adler32 (adler32-zlib) [ 2402.451518] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2402.764357] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 2404.521116] Key type lgssc registered [ 2406.040561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2407.500286] LustreError: 90159:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2424.049427] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 2428.692811] LustreError: 90159:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2434.210901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2434.236470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2435.816588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2440.216321] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2445.281514] LustreError: 91172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2445.299246] LustreError: 91172:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2448.190468] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 08:37:11 (1788439031) [ 2450.402261] LustreError: 91171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.096180] LustreError: 91203:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2452.106951] LustreError: 91203:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2452.114508] LustreError: 91203:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2452.421414] Lustre: server umount lustre-MDT0000 complete [ 2465.935258] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 2468.605934] Key type lgssc unregistered [ 2468.889836] LNet: 92695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2468.905562] LNetError: 92695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2469.989888] LNet: Removed LNI 192.168.204.108@tcp [ 2471.034628] Key type .llcrypt unregistered [ 2471.036754] Key type ._llcrypt unregistered [ 2487.585485] Key type ._llcrypt registered [ 2487.587690] Key type .llcrypt registered [ 2487.704877] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 2488.528861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2488.625654] alg: No test for adler32 (adler32-zlib) [ 2489.644515] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2489.994131] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 2491.663313] Key type lgssc registered [ 2493.024983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2501.020494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2501.066127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2502.622469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2507.683368] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2512.865489] LustreError: 93908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.882094] LustreError: 93908:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2517.988822] LustreError: 93909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2518.168984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2519.323136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2524.679248] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2533.203816] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2539.302580] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2546.105739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2546.859205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2553.703375] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2562.769230] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2616.470179] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 08:39:59 (1788439199) [ 2618.335884] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2621.237781] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 08:40:03 (1788439203) [ 2628.083160] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2628.092579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2628.111813] Lustre: Skipped 1 previous similar message [ 2633.194560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2633.197561] Lustre: Skipped 4 previous similar messages [ 2633.556607] Lustre: server umount lustre-MDT0000 complete [ 2637.923853] LustreError: 93889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439222 with bad export cookie 16081145081121162381 [ 2637.933235] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2637.945118] LustreError: 93889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2638.305067] LustreError: 94650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2638.306456] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2638.348501] Lustre: Skipped 1 previous similar message [ 2638.426956] Lustre: server umount lustre-MDT0001 complete [ 2645.207400] Lustre: server umount lustre-OST0000 complete [ 2657.920675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2658.261563] LustreError: 98208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2658.282799] LustreError: 98208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2658.481244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2663.910770] LustreError: 98209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2664.465193] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2674.146147] LustreError: 98209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2674.165766] LustreError: 98209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2674.319622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2675.104217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2680.360466] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2689.192146] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2694.660718] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2707.765209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2707.986580] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2708.000641] LustreError: 98197:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2708.037377] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2708.050467] LustreError: 100292:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2708.056232] LustreError: 100292:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2708.058507] LustreError: 100292:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2708.172092] Lustre: server umount lustre-OST0000 complete [ 2708.174837] LustreError: 100292:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2717.716182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2717.912089] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2717.923237] LustreError: 98198:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2717.949796] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2717.958414] LustreError: 100758:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2717.965853] LustreError: 100758:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2717.970288] LustreError: 100758:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2718.034885] Lustre: server umount lustre-OST0000 complete [ 2718.038793] LustreError: 100758:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2727.916710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2728.114272] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2728.124292] Lustre: Found index 0 for lustre-OST0000, updating log [ 2728.132445] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2728.225817] Lustre: lustre-OST0000: new disk, initializing [ 2728.229415] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2728.234889] Lustre: 101292:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2728.489536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2734.219867] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2738.066754] Lustre: Failing over lustre-OST0000 [ 2738.158955] Lustre: server umount lustre-OST0000 complete [ 2743.778651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2743.816210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2743.831556] Lustre: Skipped 1 previous similar message [ 2749.922068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2749.942201] Lustre: Skipped 3 previous similar messages [ 2753.373282] Lustre: server umount lustre-MDT0000 complete [ 2755.043638] LustreError: 98208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2755.096108] LustreError: 98208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2758.801328] LustreError: 100311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439343 with bad export cookie 16081145081121165125 [ 2758.817437] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2758.826042] LustreError: 100311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2759.482734] Lustre: server umount lustre-MDT0001 complete [ 2771.441068] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 08:42:34 (1788439354) [ 2788.999531] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 2800.478759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2801.133441] LustreError: 103637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2801.282497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2806.369625] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2817.115718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2823.192045] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2826.749704] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2834.943652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2836.265434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2836.273650] Lustre: Skipped 1 previous similar message [ 2836.284712] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2842.421338] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2842.430292] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2842.456584] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2844.437158] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2846.573356] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:05 [ 2846.595760] Lustre: lustre-OST0000: Recovery over after 0:04, of 1 clients 1 recovered and 0 were evicted. [ 2846.596077] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2846.603495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2846.632358] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2856.347133] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2856.472224] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2856.588948] Lustre: lustre-OST0001: new disk, initializing [ 2856.593134] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2856.606124] Lustre: 105962:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2862.228390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2862.237038] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2862.400826] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2864.201857] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2874.624758] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2883.963168] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2889.700803] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2889.707532] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2889.724238] Lustre: Skipped 1 previous similar message [ 2889.745876] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2895.922929] Lustre: server umount lustre-MDT0001 complete [ 2897.752657] LustreError: 103634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2897.802591] LustreError: 103634:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2910.344268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2910.514374] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2910.528085] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2910.652478] Lustre: 103626:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2911.664941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2911.670585] Lustre: Skipped 1 previous similar message [ 2911.974074] LustreError: 107485:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2911.987654] LustreError: 107485:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2912.010905] LustreError: 103629:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2912.176887] LustreError: 107500:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.108@tcp: cfg command failed: rc = -17 [ 2912.208237] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2912.208237] [ 2916.854549] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2917.611326] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 2931.287568] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 08:45:13 (1788439513) [ 2956.044312] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 2977.808521] Lustre: 109598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3007.278981] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3012.871994] Lustre: 110734:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3114.297536] Lustre: server umount lustre-MDT0000 complete [ 3116.002037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3116.011714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3116.027023] Lustre: Skipped 2 previous similar messages [ 3116.037428] LustreError: 103637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3116.064509] LustreError: 103637:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 3118.542464] LustreError: 103619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439703 with bad export cookie 16081145081121166126 [ 3118.546553] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3118.553453] LustreError: 103619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3118.942437] Lustre: server umount lustre-MDT0001 complete [ 3134.244847] Lustre: server umount lustre-OST0000 complete [ 3137.892071] Lustre: 93221:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788439706/real 1788439706] req@ffff9fc84334ca80 x1875314290842496/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788439722 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3137.927226] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3137.947895] Lustre: Skipped 3 previous similar messages [ 3138.943351] Lustre: server umount lustre-OST0001 complete [ 3145.692340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3158.677738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3169.965421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3182.199522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3207.342324] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 3220.935624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3220.951805] Lustre: lustre-MDT0000: reset Object Index mappings [ 3221.567243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3226.897427] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3236.589778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3236.625640] Lustre: lustre-MDT0001: reset Object Index mappings [ 3243.070635] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3246.964795] Lustre: 115839:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3255.888982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3261.234524] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3264.150799] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3266.354264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3276.128347] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3282.428740] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3282.447170] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3284.806893] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3295.912428] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3301.367166] Lustre: 117718:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3366.496352] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 08:52:29 (1788439949) [ 3395.046781] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3395.050141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3395.063567] Lustre: Skipped 2 previous similar messages [ 3395.074705] Lustre: Skipped 7 previous similar messages [ 3399.323456] Lustre: server umount lustre-MDT0000 complete [ 3400.170421] LustreError: 114690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3400.190459] LustreError: 114690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 3403.498920] LustreError: 114672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439988 with bad export cookie 16081145081121213215 [ 3403.505894] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3403.509336] LustreError: 114672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3409.940255] Lustre: server umount lustre-MDT0001 complete [ 3428.320627] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3428.539319] Lustre: server umount lustre-OST0000 complete [ 3447.263237] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3447.558870] Lustre: server umount lustre-OST0001 complete [ 3455.564165] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 3467.281638] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 3511.736108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3512.014695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3512.042234] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3512.106798] Lustre: lustre-MDT0000: new disk, initializing [ 3512.184919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3512.193515] Lustre: Skipped 3 previous similar messages [ 3512.208965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3517.835725] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3526.017245] Lustre: server umount lustre-MDT0000 complete [ 3538.078217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3538.633763] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3538.671160] Lustre: 121995:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3538.720800] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3545.205913] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3551.428670] Lustre: server umount lustre-MDT0000 complete [ 3563.845597] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 08:55:46 (1788440146) [ 3566.029669] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3568.392588] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 08:55:51 (1788440151) [ 3590.207584] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 3601.359312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3607.531234] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3617.738728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3617.918162] Lustre: 124048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3617.989860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3618.012438] Lustre: Skipped 1 previous similar message [ 3618.106394] Lustre: lustre-MDT0001: new disk, initializing [ 3618.301604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3618.322702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3624.317723] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3629.225722] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3636.931330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3637.269335] Lustre: lustre-OST0000: new disk, initializing [ 3637.272444] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3637.278356] Lustre: 125626:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3638.980380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3639.008876] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3639.154687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3645.144644] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3658.106287] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3658.326580] Lustre: lustre-OST0001: new disk, initializing [ 3658.332279] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3658.343466] Lustre: 126497:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3659.825655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3659.862388] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3660.083956] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3666.728400] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 3677.475962] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3681.511930] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4135.113676] Lustre: 132932:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4135.125382] Lustre: 132932:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4135.133579] Lustre: 132932:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4135.152153] Lustre: 132932:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4135.166554] Lustre: 132932:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4135.182165] Lustre: 132932:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4486.270299] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:11:08 (1788441068) [ 4504.032867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4504.045939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4504.069711] Lustre: Skipped 3 previous similar messages [ 4504.080597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4504.092335] Lustre: Skipped 4 previous similar messages [ 4508.763740] Lustre: server umount lustre-MDT0000 complete [ 4510.181384] LustreError: 127294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4510.197680] LustreError: 127294:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4512.994183] LustreError: 124041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441097 with bad export cookie 16081145081121285140 [ 4513.005883] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4513.012461] LustreError: 124041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4515.299719] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4515.302937] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4515.325566] Lustre: Skipped 4 previous similar messages [ 4515.350055] Lustre: Skipped 5 previous similar messages [ 4519.846810] Lustre: server umount lustre-MDT0001 complete [ 4530.485471] Lustre: server umount lustre-OST0000 complete [ 4541.577703] Lustre: server umount lustre-OST0001 complete [ 4550.145039] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 4562.047866] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 4604.006575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4604.323449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4604.369560] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4604.488162] Lustre: lustre-MDT0000: new disk, initializing [ 4604.649537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4604.656377] Lustre: Skipped 5 previous similar messages [ 4604.684040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4609.903921] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4623.143752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4623.270967] Lustre: 136001:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4623.326754] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4623.332401] Lustre: Skipped 1 previous similar message [ 4623.392355] Lustre: lustre-MDT0001: new disk, initializing [ 4623.513045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4623.525741] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4628.645488] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4642.880136] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4651.045309] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4658.330887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4658.673100] Lustre: lustre-OST0000: new disk, initializing [ 4658.681290] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4658.689963] Lustre: 137969:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4659.860702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4659.869674] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4659.948545] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4664.440327] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 4676.717446] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4691.836478] Lustre: lustre-MDD0000: changelog on [ 4695.379342] Lustre: lustre-MDD0001: changelog on [ 6957.804752] Lustre: lustre-MDD0001: changelog off [ 6960.140197] Lustre: 136007:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6961.002550] Lustre: lustre-MDD0000: changelog off [ 6969.312792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6969.319833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6969.321726] LustreError: Skipped 1 previous similar message [ 6969.332865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6969.371869] Lustre: Skipped 1 previous similar message [ 6969.389804] Lustre: Skipped 3 previous similar messages [ 6972.387562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6972.402548] Lustre: Skipped 1 previous similar message [ 6974.630115] Lustre: server umount lustre-OST0000 complete [ 6982.628588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6982.655973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6982.664355] Lustre: Skipped 1 previous similar message [ 6985.052550] Lustre: server umount lustre-MDT0000 complete [ 6987.755155] LustreError: 136013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6987.778101] LustreError: 136013:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6989.507951] LustreError: 135993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443574 with bad export cookie 16081145081122111315 [ 6989.517411] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6989.520123] LustreError: 135993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6989.926167] Lustre: server umount lustre-MDT0001 complete [ 7001.533989] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 7004.537723] Key type lgssc unregistered [ 7004.829634] LNet: 147162:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7004.848575] LNetError: 147162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7004.870804] LNet: Removed LNI 192.168.204.108@tcp [ 7005.779736] Key type .llcrypt unregistered [ 7005.783750] Key type ._llcrypt unregistered [ 7028.783957] Key type ._llcrypt registered [ 7028.788682] Key type .llcrypt registered [ 7028.951384] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 7045.310550] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 7046.637619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7046.770689] alg: No test for adler32 (adler32-zlib) [ 7048.013287] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7048.285125] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 7049.944198] Key type lgssc registered [ 7051.368450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7088.994303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7089.052943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7090.463667] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7090.491611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7090.585378] Lustre: lustre-MDT0000: new disk, initializing [ 7090.711960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7090.737187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7095.434371] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7106.379700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7106.481561] Lustre: 150361:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7106.562513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7106.571520] Lustre: Skipped 1 previous similar message [ 7106.727992] Lustre: lustre-MDT0001: new disk, initializing [ 7106.820476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7106.855432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7106.864199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7113.121951] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7124.941784] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7131.816468] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7139.264337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7139.744230] Lustre: lustre-OST0000: new disk, initializing [ 7139.754909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7139.762399] Lustre: 152330:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7139.907869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7145.930760] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7148.652168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7148.668797] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7148.785570] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7156.493214] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7162.993240] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_module obdecho/obdecho [ 7165.368435] LustreError: 153333:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7165.375407] LustreError: 153333:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7168.099928] LustreError: 153429:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7183.847372] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7183.853550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7183.861888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7184.868855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7184.875923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7184.884229] Lustre: Skipped 1 previous similar message [ 7184.887735] Lustre: Skipped 1 previous similar message [ 7186.589414] Lustre: server umount lustre-MDT0000 complete [ 7190.256498] LustreError: 150354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443775 with bad export cookie 13908694359360101640 [ 7190.270640] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7190.283270] LustreError: 150354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7190.527818] Lustre: server umount lustre-MDT0001 complete [ 7205.507993] Lustre: server umount lustre-OST0000 complete [ 7230.165734] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 7244.270653] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 7289.099492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7289.331671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7289.365262] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7289.475613] Lustre: lustre-MDT0000: new disk, initializing [ 7289.548750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7289.567686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7294.907720] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7307.983223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7308.080660] Lustre: 157163:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7308.150768] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7308.153720] Lustre: Skipped 1 previous similar message [ 7308.214316] Lustre: lustre-MDT0001: new disk, initializing [ 7308.337941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7308.390047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7308.408619] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7314.484193] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7326.452530] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7334.021888] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7342.789821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7343.072752] Lustre: lustre-OST0000: new disk, initializing [ 7343.086785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7343.093387] Lustre: 159131:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7343.171357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7350.595344] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7351.355487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7351.367647] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7351.465159] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7362.977295] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7368.446611] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7368.724232] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7375.153860] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7375.438565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7382.498206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7382.506872] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7382.561310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7382.573713] LustreError: Skipped 1 previous similar message [ 7382.590786] Lustre: Skipped 2 previous similar messages [ 7387.618838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7387.633879] Lustre: Skipped 1 previous similar message [ 7392.223830] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7392.688821] Lustre: server umount lustre-OST0000 complete [ 7397.864823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7397.872513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7397.885919] Lustre: Skipped 2 previous similar messages [ 7397.899547] Lustre: Skipped 1 previous similar message [ 7403.860402] Lustre: server umount lustre-MDT0000 complete [ 7408.098477] LustreError: 157169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7408.132260] LustreError: 157169:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7409.142078] LustreError: 157152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443994 with bad export cookie 13908694359360102599 [ 7409.144744] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7409.159067] LustreError: 157152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7409.695362] Lustre: server umount lustre-MDT0001 complete [ 7414.683963] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 09:59:57 (1788443997) [ 7416.183701] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7418.227885] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:00:01 (1788444001) [ 7426.753212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7427.179140] LustreError: 161545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7427.262317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7432.392483] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7432.673233] LustreError: 161546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7437.794721] LustreError: 161545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7442.919805] LustreError: 161546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7443.012986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7448.075652] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7457.600794] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7464.353274] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7472.226694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7472.559962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7472.570413] Lustre: Skipped 1 previous similar message [ 7478.154336] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7488.808483] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7499.746961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7499.749703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7499.798539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7499.818959] Lustre: Skipped 2 previous similar messages [ 7501.439274] Lustre: server umount lustre-MDT0000 complete [ 7504.364295] LustreError: 161546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7504.401566] LustreError: 161546:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7505.918657] LustreError: 161527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444090 with bad export cookie 13908694359360103488 [ 7505.925556] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7505.933021] LustreError: 161527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7509.473533] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7509.485658] Lustre: Skipped 2 previous similar messages [ 7515.975365] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.8@tcp (stopping) [ 7515.987592] Lustre: Skipped 5 previous similar messages [ 7520.224039] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7520.744732] LustreError: 161540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7520.768327] LustreError: 161540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 7521.007405] Lustre: server umount lustre-MDT0001 complete [ 7531.153647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7546.853147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7547.185196] LustreError: 165400:0:(llog.c:1655:llog_backup()) MGC192.168.204.108@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7547.199291] Lustre: 165400:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.108@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7555.553314] LustreError: 165405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7555.582717] LustreError: 165405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7608.288914] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7608.314824] Lustre: 165400:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.108@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7621.088725] LustreError: 165406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7621.117375] LustreError: 165406:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 46 previous similar messages [ 7648.767160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7650.797946] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7654.360127] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7664.245419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7675.378484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7681.041613] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7681.286328] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7691.323337] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7693.204563] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7699.734738] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7701.356977] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7721.953978] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7721.958515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7721.960977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7721.980448] Lustre: Skipped 1 previous similar message [ 7723.785694] Lustre: server umount lustre-MDT0000 complete [ 7728.357908] LustreError: 161525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444313 with bad export cookie 13908694359360109977 [ 7728.359610] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7728.376302] LustreError: 161525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7728.877476] Lustre: server umount lustre-MDT0001 complete [ 7745.273523] Lustre: server umount lustre-OST0000 complete [ 7769.212279] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 7785.868042] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 7832.352772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7832.661725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7832.711285] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7832.868200] Lustre: lustre-MDT0000: new disk, initializing [ 7833.003325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7833.010649] Lustre: Skipped 1 previous similar message [ 7833.048776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7839.088218] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7850.535770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7850.688148] Lustre: 170531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7851.027418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7856.913478] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7868.786432] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7875.837341] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7883.969834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7884.159666] Lustre: lustre-OST0000: new disk, initializing [ 7884.162527] Lustre: Skipped 1 previous similar message [ 7884.166642] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7884.170420] Lustre: Skipped 2 previous similar messages [ 7884.176951] Lustre: 172499:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7891.050959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7891.077058] Lustre: Skipped 1 previous similar message [ 7891.091909] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7891.227117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7891.274967] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7901.507572] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7908.002891] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7908.232985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7912.996160] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7913.317556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7917.032702] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7917.040427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7917.062376] Lustre: Skipped 1 previous similar message [ 7917.067567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7917.073601] Lustre: Skipped 2 previous similar messages [ 7921.572225] Lustre: server umount lustre-OST0000 complete [ 7932.117595] Lustre: server umount lustre-MDT0000 complete [ 7932.398694] LustreError: 170538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7932.430768] LustreError: 170538:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 46 previous similar messages [ 7936.465208] LustreError: 170524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444521 with bad export cookie 13908694359360126777 [ 7936.478312] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7936.480235] LustreError: 170524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7937.177602] Lustre: server umount lustre-MDT0001 complete [ 7949.338823] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:08:51 (1788444531) [ 7957.714573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7964.704316] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7974.936782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7980.797428] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 7990.250722] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7997.249913] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8005.917826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8013.615864] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8022.703798] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8038.910019] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:10:21 (1788444621) [ 8044.514488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8044.516407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8044.542249] Lustre: Skipped 2 previous similar messages [ 8044.547898] LustreError: Skipped 1 previous similar message [ 8049.730914] Lustre: server umount lustre-OST0000 complete [ 8054.759961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8054.774565] Lustre: Skipped 6 previous similar messages [ 8061.022332] Lustre: server umount lustre-MDT0000 complete [ 8065.435673] LustreError: 174893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444650 with bad export cookie 13908694359360127659 [ 8065.449715] LustreError: 174893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8065.450682] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8065.911892] Lustre: server umount lustre-MDT0001 complete [ 8081.160960] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 8085.295787] Key type lgssc unregistered [ 8085.574682] LNet: 178651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8085.584193] LNetError: 178651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8085.601546] LNet: Removed LNI 192.168.204.108@tcp [ 8086.366263] Key type .llcrypt unregistered [ 8086.367968] Key type ._llcrypt unregistered [ 8111.637656] Key type ._llcrypt registered [ 8111.639990] Key type .llcrypt registered [ 8111.744476] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 8113.163940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8113.334212] alg: No test for adler32 (adler32-zlib) [ 8114.393177] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8114.652686] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 8116.367314] Key type lgssc registered [ 8117.461722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8126.417972] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8126.445336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8142.111539] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8147.231363] LustreError: 179895:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.108@tcp: failed processing log, type 4: rc = -110 [ 8152.352462] LustreError: 179895:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8152.372523] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8152.378871] LustreError: MGC192.168.204.108@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8152.385859] LustreError: 179895:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8152.398727] LustreError: 179895:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 8152.410490] LustreError: 179895:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8152.455719] Lustre: server umount lustre-OST0000 complete [ 8152.458959] LustreError: 179895:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8163.339082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8164.716108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8169.874550] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8175.082104] LustreError: 180394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8175.100794] LustreError: 180394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8179.929155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8180.279660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8185.147354] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8194.421837] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8201.751748] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8209.672217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8209.821576] LustreError: 182212:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8209.829668] Lustre: 182212:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.108@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8209.837151] LustreError: 182217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8210.051837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8217.064559] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8226.341155] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8230.886058] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8230.894789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8230.912789] Lustre: Skipped 2 previous similar messages [ 8230.927040] Lustre: Skipped 2 previous similar messages [ 8234.879654] Lustre: server umount lustre-MDT0000 complete [ 8236.008351] LustreError: 180393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8236.057766] LustreError: 180393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8239.011406] LustreError: 181386:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8239.071794] LustreError: 181386:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8240.201723] LustreError: 181128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444825 with bad export cookie 17327187118354370527 [ 8240.203785] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8240.214642] LustreError: 181128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8240.713386] Lustre: server umount lustre-MDT0001 complete [ 8256.184751] Lustre: server umount lustre-OST0000 complete [ 8269.587782] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 8282.825722] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 8325.548723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8325.938049] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8326.015665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8326.148975] Lustre: lustre-MDT0000: new disk, initializing [ 8326.293844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8326.325380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8331.563579] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8344.237284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8344.421017] Lustre: 185751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8344.506177] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8344.509475] Lustre: Skipped 1 previous similar message [ 8344.559459] Lustre: lustre-MDT0001: new disk, initializing [ 8344.644874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8344.675241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8344.681109] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8349.938212] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8361.708961] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8369.065640] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8378.158586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8378.709557] Lustre: lustre-OST0000: new disk, initializing [ 8378.726559] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8378.742903] Lustre: 187722:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8378.938177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8387.212765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8387.231939] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8387.456081] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8388.940787] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8401.145677] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8407.958366] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8408.417885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8415.319605] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8415.606774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8418.272509] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8418.280075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8418.285595] LustreError: Skipped 1 previous similar message [ 8418.286295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8418.319183] Lustre: Skipped 1 previous similar message [ 8423.399349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8423.410734] Lustre: Skipped 2 previous similar messages [ 8424.398581] Lustre: server umount lustre-OST0000 complete [ 8431.586365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8431.590994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8431.600821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8433.632978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8435.390185] Lustre: server umount lustre-MDT0000 complete [ 8438.759662] LustreError: 186929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8438.783397] LustreError: 186929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8440.372309] LustreError: 185745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445025 with bad export cookie 17327187118354371248 [ 8440.377891] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8440.385484] LustreError: 185745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8441.051990] Lustre: server umount lustre-MDT0001 complete [ 8451.842792] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 10:17:14 (1788445034) [ 8472.585071] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 8476.028448] Key type lgssc unregistered [ 8476.427297] LNet: 190671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8476.438136] LNetError: 190671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8477.480358] LNet: Removed LNI 192.168.204.108@tcp [ 8478.639422] Key type .llcrypt unregistered [ 8478.647124] Key type ._llcrypt unregistered [ 8499.375744] Key type ._llcrypt registered [ 8499.377569] Key type .llcrypt registered [ 8499.502324] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 8500.618423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8500.688494] alg: No test for adler32 (adler32-zlib) [ 8502.105072] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8502.536221] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 8504.255489] Key type lgssc registered [ 8505.612270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8514.026805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8514.039745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8515.550369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8521.126587] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8525.806488] LustreError: 191882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8525.832876] LustreError: 191882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8530.914494] LustreError: 191883:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8531.110048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8531.486736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8537.847545] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8547.092511] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8554.311529] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8563.867045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8564.194327] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8564.197521] LustreError: 193748:0:(llog.c:1696:llog_backup()) MGC192.168.204.108@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8564.204369] Lustre: 193748:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.108@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8564.356547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8571.762735] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8581.039535] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8587.233991] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8587.242170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8587.262400] Lustre: Skipped 2 previous similar messages [ 8587.283772] Lustre: Skipped 2 previous similar messages [ 8590.033459] Lustre: server umount lustre-MDT0000 complete [ 8592.357436] LustreError: 191877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8592.386939] LustreError: 191877:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8594.045092] LustreError: 192618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445178 with bad export cookie 1477255420050022762 [ 8594.055532] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8594.066288] LustreError: 192618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8594.404756] Lustre: server umount lustre-MDT0001 complete [ 8608.797906] Lustre: server umount lustre-OST0000 complete [ 8620.249978] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 8632.731340] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 8677.414926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8677.733524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8677.781988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8677.919326] Lustre: lustre-MDT0000: new disk, initializing [ 8678.031704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8678.065221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8683.906616] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8695.171249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8695.290233] Lustre: 197275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8695.341345] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8695.343151] Lustre: Skipped 1 previous similar message [ 8695.346018] LustreError: 198058:0:(llog.c:1696:llog_backup()) MGC192.168.204.108@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8695.351093] LustreError: 198058:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8695.357348] Lustre: 198058:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.108@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8695.376733] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8695.380040] Lustre: Skipped 1 previous similar message [ 8695.478673] Lustre: lustre-MDT0001: new disk, initializing [ 8695.633781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8695.718669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8695.728964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8700.548234] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8709.388793] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8715.112626] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8723.394388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8723.713728] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8723.717210] Lustre: Skipped 1 previous similar message [ 8723.720321] LustreError: 199219:0:(llog.c:1696:llog_backup()) MGC192.168.204.108@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8723.726357] LustreError: 199219:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8723.732509] Lustre: 199219:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.108@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8723.737691] Lustre: 199219:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8723.758678] Lustre: lustre-OST0000: new disk, initializing [ 8723.762169] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8723.770486] Lustre: 199242:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8723.864983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8730.218364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8730.235054] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8730.402972] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8730.772751] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8741.785349] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8747.835402] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8748.259949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8754.457430] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8754.727962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8761.314306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8761.316729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8761.324993] LustreError: Skipped 1 previous similar message [ 8761.337942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8761.362665] Lustre: Skipped 1 previous similar message [ 8763.228565] Lustre: server umount lustre-OST0000 complete [ 8767.457467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8767.468974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8767.487040] Lustre: Skipped 1 previous similar message [ 8767.501193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8771.553382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8771.572435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8773.577549] Lustre: server umount lustre-MDT0000 complete [ 8776.676373] LustreError: 198692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8776.702665] LustreError: 198692:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8778.439103] LustreError: 198079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445363 with bad export cookie 1477255420050023476 [ 8778.440232] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8778.461651] LustreError: 198079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8778.952701] Lustre: server umount lustre-MDT0001 complete [ 8789.729928] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:22:52 (1788445372) [ 8815.118787] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 8828.179121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8828.802266] LustreError: 202396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8828.863948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8833.292247] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8838.121563] LustreError: 202397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8838.147199] LustreError: 202397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8843.360449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8848.484549] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8851.847403] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8860.712053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8861.234546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8861.244741] Lustre: Skipped 1 previous similar message [ 8870.428830] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8880.802787] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8881.022499] Lustre: lustre-OST0001: new disk, initializing [ 8881.027863] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8881.033786] Lustre: 204707:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8888.632701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8888.656698] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8888.887914] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8891.162113] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8903.795938] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8911.907933] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8914.694982] Lustre: DEBUG MARKER: ADD OST3 [ 8920.708624] loop: module loaded [ 8920.821360] loop0: detected capacity change from 0 to 204800000 [ 8921.044135] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8921.056710] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8921.067920] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8929.018389] Lustre: DEBUG MARKER: STOP OST3 [ 8929.597069] loop0: detected capacity change from 0 to 204800000 [ 8936.570404] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8936.891512] Lustre: lustre-OST0003: new disk, initializing [ 8936.900919] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8936.910847] Lustre: 206792:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8937.108578] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8937.114889] Lustre: Skipped 1 previous similar message [ 8938.705283] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8938.912642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8938.922501] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8938.982958] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8939.608653] Lustre: Failing over lustre-OST0003 [ 8939.946885] Lustre: server umount lustre-OST0003 complete [ 8940.774314] LustreError: 206930:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8943.477141] LustreError: 203884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8943.504385] LustreError: 203884:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8944.138141] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8944.138437] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8944.156216] Lustre: Skipped 1 previous similar message [ 8945.456668] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8957.980603] Lustre: DEBUG MARKER: START OST3 again [ 8966.452067] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8966.869813] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8968.128979] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8968.458876] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8968.461549] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8975.128647] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 8997.712071] Lustre: Failing over lustre-OST0003 [ 8997.824698] Lustre: server umount lustre-OST0003 complete [ 8997.891876] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8997.899137] LustreError: 203883:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8997.916926] Lustre: Skipped 1 previous similar message [ 8997.944122] LustreError: 203883:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 9017.827137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9017.842344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9017.855399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9017.857483] Lustre: Skipped 1 previous similar message [ 9022.506306] Lustre: server umount lustre-MDT0000 complete [ 9027.005842] LustreError: 204706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445611 with bad export cookie 1477255420050024358 [ 9027.020636] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9027.030781] LustreError: 204706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9028.603348] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9028.620322] Lustre: Skipped 5 previous similar messages [ 9041.378129] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9041.846438] Lustre: server umount lustre-MDT0001 complete [ 9047.698088] Lustre: server umount lustre-OST0000 complete [ 9052.780932] Lustre: server umount lustre-OST0001 complete [ 9063.587212] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 9075.156863] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 9122.448763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9122.719326] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9122.741220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9122.817750] Lustre: lustre-MDT0000: new disk, initializing [ 9122.880526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9122.885732] Lustre: Skipped 1 previous similar message [ 9122.900822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9128.314819] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9141.581938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9141.709205] Lustre: 212332:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9141.814640] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9141.825370] Lustre: Skipped 1 previous similar message [ 9141.973172] Lustre: lustre-MDT0001: new disk, initializing [ 9142.165534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9142.190470] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9147.675458] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9160.275739] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9168.278057] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9176.845766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9177.273623] Lustre: lustre-OST0000: new disk, initializing [ 9177.279282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9177.285411] Lustre: 214299:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9179.019197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9179.030761] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9179.253384] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9185.200763] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9196.702505] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9202.021665] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9202.299950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9208.520522] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9208.794478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9214.947945] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9214.951020] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9214.959570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9214.959578] Lustre: Skipped 4 previous similar messages [ 9214.968445] Lustre: Skipped 6 previous similar messages [ 9225.183199] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9225.304201] Lustre: server umount lustre-OST0000 complete [ 9236.006546] Lustre: server umount lustre-MDT0000 complete [ 9240.558435] LustreError: 212338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9240.591388] LustreError: 212338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 9240.718210] LustreError: 215843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445825 with bad export cookie 1477255420050026654 [ 9240.728314] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9240.742967] LustreError: 215843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9241.303801] Lustre: server umount lustre-MDT0001 complete [ 9246.347844] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 10:30:28 (1788445828) [ 9269.177239] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 9281.978731] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 9322.853623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9323.078631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9323.097595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9323.159882] Lustre: lustre-MDT0000: new disk, initializing [ 9323.216148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9327.693390] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9343.532368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9343.784338] Lustre: 218918:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9344.252980] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9351.477963] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9363.804217] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9371.708995] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9379.200403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9379.528713] Lustre: 220889:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9379.651381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9379.658796] Lustre: Skipped 4 previous similar messages [ 9380.905796] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9381.154699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9385.908795] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9395.392787] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9400.508435] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9400.757769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9406.311141] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9406.563356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9412.066818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9412.069562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9412.081367] LustreError: Skipped 1 previous similar message [ 9412.089809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9412.095508] Lustre: Skipped 3 previous similar messages [ 9412.111683] Lustre: Skipped 10 previous similar messages [ 9414.906797] Lustre: server umount lustre-OST0000 complete [ 9425.256906] Lustre: server umount lustre-MDT0000 complete [ 9427.427906] LustreError: 218929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9427.456635] LustreError: 218929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9429.035653] LustreError: 218910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446013 with bad export cookie 1477255420050027536 [ 9429.045947] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9429.049613] LustreError: 218910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9429.403520] Lustre: server umount lustre-MDT0001 complete [ 9438.184038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9443.989968] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9453.068384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9457.911209] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9468.191264] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9478.512268] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9490.215450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9498.464638] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9507.365532] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9519.585093] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9519.590740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9519.596085] LustreError: Skipped 2 previous similar messages [ 9519.607508] Lustre: Skipped 4 previous similar messages [ 9524.031978] Lustre: server umount lustre-OST0000 complete [ 9543.139566] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9543.540777] Lustre: server umount lustre-MDT0000 complete [ 9548.797699] LustreError: 224951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446133 with bad export cookie 1477255420050028411 [ 9548.802404] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9548.819621] LustreError: 224951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9549.467533] Lustre: server umount lustre-MDT0001 complete [ 9650.106245] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 9653.377875] Key type lgssc unregistered [ 9653.727303] LNet: 226771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9653.739930] LNetError: 226771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9653.763852] LNet: Removed LNI 192.168.204.108@tcp [ 9654.893510] Key type .llcrypt unregistered [ 9654.896185] Key type ._llcrypt unregistered [ 9683.724916] Key type ._llcrypt registered [ 9683.730837] Key type .llcrypt registered [ 9683.889462] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 9699.833347] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 9701.109937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9701.127345] alg: No test for adler32 (adler32-zlib) [ 9702.218926] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9702.478271] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 9704.159248] Key type lgssc registered [ 9705.431059] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9743.424191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9743.458190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9744.720256] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9744.754116] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9744.820268] Lustre: lustre-MDT0000: new disk, initializing [ 9744.936247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9744.959764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9749.528507] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9760.668822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9760.850239] Lustre: 230121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9760.925460] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9760.930277] Lustre: Skipped 1 previous similar message [ 9761.015574] Lustre: lustre-MDT0001: new disk, initializing [ 9761.110486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9761.152307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9761.157808] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9765.838918] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9776.180206] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9781.917093] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9788.956230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9789.272627] Lustre: lustre-OST0000: new disk, initializing [ 9789.277550] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9789.285976] Lustre: 232092:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9789.391815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9795.855730] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9799.771275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9799.792523] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9799.926460] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9807.062914] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9812.618811] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9812.874365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9818.336992] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9818.576401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9825.763109] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9825.766236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9825.778325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9825.804945] Lustre: Skipped 1 previous similar message [ 9830.885572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9830.893542] Lustre: Skipped 1 previous similar message [ 9834.975329] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9835.149343] Lustre: server umount lustre-OST0000 complete [ 9841.121718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9841.127778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9841.131730] Lustre: Skipped 1 previous similar message [ 9841.141742] Lustre: Skipped 1 previous similar message [ 9846.241781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9846.251480] Lustre: Skipped 1 previous similar message [ 9851.360116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9851.367947] Lustre: Skipped 1 previous similar message [ 9853.407824] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9853.596538] Lustre: server umount lustre-MDT0000 complete [ 9857.458237] LustreError: 230113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446442 with bad export cookie 6280233664326872562 [ 9857.469814] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9857.471453] LustreError: 230113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9857.834635] Lustre: server umount lustre-MDT0001 complete [ 9876.569962] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 9888.390423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9888.902594] LustreError: 235026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9888.924740] LustreError: 235026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9888.965170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9893.946728] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9894.372142] LustreError: 235027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9898.481534] LustreError: 235026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9903.280641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9903.598579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9907.888653] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9911.123817] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9918.590160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9919.003709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9924.961896] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9934.195194] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9934.394687] Lustre: lustre-OST0001: new disk, initializing [ 9934.403156] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9934.421742] Lustre: 237335:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9937.433295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9937.443920] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9937.586148] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9940.796881] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [ 9953.116868] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9963.655387] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9977.887592] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 10:42:40 (1788446560) [ 9980.384366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9980.387911] LustreError: Skipped 1 previous similar message [ 9980.392546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9980.402077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9983.974036] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9983.999871] Lustre: Skipped 2 previous similar messages [ 9986.530892] Lustre: server umount lustre-MDT0000 complete [ 9989.099551] LustreError: 235027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9989.124145] LustreError: 235027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9990.990495] LustreError: 238104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446575 with bad export cookie 6280233664326873437 [ 9990.998570] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9991.001630] LustreError: 238104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9991.518347] Lustre: server umount lustre-MDT0001 complete [10006.152768] Lustre: server umount lustre-OST0000 complete [10019.986455] Lustre: server umount lustre-OST0001 complete [10026.115928] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [10034.575904] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [10068.949235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10069.201643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10069.240056] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10069.304976] Lustre: lustre-MDT0000: new disk, initializing [10069.376922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10069.384919] Lustre: Skipped 1 previous similar message [10069.401871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10073.587692] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10084.197329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10084.288802] Lustre: 241284:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10084.330845] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10084.333890] Lustre: Skipped 1 previous similar message [10084.407898] Lustre: lustre-MDT0001: new disk, initializing [10084.543320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10084.556761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10089.330126] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10099.180525] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10105.750945] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10113.385108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10113.707312] Lustre: lustre-OST0000: new disk, initializing [10113.715557] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10113.726342] Lustre: 243255:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10113.840618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10113.857597] Lustre: Skipped 1 previous similar message [10115.440430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10115.446551] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10115.590115] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10120.338690] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10130.530589] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10136.015497] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10136.346407] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10141.187596] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10141.437637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10146.290256] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10146.296679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10146.296775] LustreError: Skipped 1 previous similar message [10146.303915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10146.316762] Lustre: Skipped 1 previous similar message [10146.343496] Lustre: Skipped 5 previous similar messages [10149.640662] Lustre: server umount lustre-OST0000 complete [10156.516499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10156.520286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10159.643979] Lustre: server umount lustre-MDT0000 complete [10161.643717] LustreError: 241297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10161.670567] LustreError: 241297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10163.752512] LustreError: 241277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446748 with bad export cookie 6280233664326875579 [10163.769379] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10163.773090] LustreError: 241277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10164.162666] Lustre: server umount lustre-MDT0001 complete [10172.864899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10173.403147] LustreError: 245474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10178.299276] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10186.201731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10186.614476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10186.620968] Lustre: Skipped 1 previous similar message [10190.902303] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10198.871524] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10204.255624] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10210.704644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10217.599491] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10226.257735] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10376.164958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10376.168297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10376.177509] Lustre: Skipped 1 previous similar message [10376.182677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10376.188262] Lustre: Skipped 2 previous similar messages [10379.660114] Lustre: server umount lustre-OST0000 complete [10386.401900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10386.418113] LustreError: Skipped 1 previous similar message [10389.876943] Lustre: server umount lustre-MDT0000 complete [10392.544846] LustreError: 247971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10392.554066] LustreError: 247971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10393.666968] LustreError: 245456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446978 with bad export cookie 6280233664326876461 [10393.676813] LustreError: 245456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10393.679540] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10394.070337] Lustre: server umount lustre-MDT0001 complete [10497.750831] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [10500.493406] Key type lgssc unregistered [10500.757614] LNet: 249090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10500.768841] LNetError: 249090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10500.787716] LNet: Removed LNI 192.168.204.108@tcp [10501.720173] Key type .llcrypt unregistered [10501.723333] Key type ._llcrypt unregistered [10528.777289] Key type ._llcrypt registered [10528.779317] Key type .llcrypt registered [10528.881426] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [10544.407515] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [10545.381794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10545.581680] alg: No test for adler32 (adler32-zlib) [10546.699985] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10547.033445] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [10548.696378] Key type lgssc registered [10549.816456] Lustre: Echo OBD driver; http://www.lustre.org/ [10596.818267] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [10607.359966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10607.392140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10608.825753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10608.893186] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10608.991710] Lustre: lustre-MDT0000: new disk, initializing [10609.078232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10609.094949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10614.341820] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10625.722591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10625.818223] Lustre: 253157:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10625.917829] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10625.922324] Lustre: Skipped 1 previous similar message [10626.026371] Lustre: lustre-MDT0001: new disk, initializing [10626.223947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10626.285066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10626.311067] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10631.655593] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10637.055522] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10644.173894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10644.425928] Lustre: lustre-OST0000: new disk, initializing [10644.431678] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10644.438730] Lustre: 254788:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10644.560779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10651.081856] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10651.226506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10651.236250] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10651.391168] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10661.466582] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10661.641457] Lustre: lustre-OST0001: new disk, initializing [10661.645623] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10661.650841] Lustre: 255660:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10661.772124] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10667.535466] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10668.661931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10668.677468] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10668.805829] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10678.319981] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10686.730772] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10697.696338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10697.714526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10697.732350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10699.625583] Lustre: server umount lustre-MDT0000 complete [10699.752307] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10703.362134] LustreError: 253148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447288 with bad export cookie 11288942472903290784 [10703.365104] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10703.371870] LustreError: 253148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10703.705168] Lustre: server umount lustre-MDT0001 complete [10717.810995] Lustre: server umount lustre-OST0000 complete [10720.736486] Lustre: 250665:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788447289/real 1788447289] req@ffff9fc981c84000 x1875322738910208/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788447305 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10720.762505] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10720.779153] Lustre: Skipped 2 previous similar messages [10721.600522] Lustre: server umount lustre-OST0001 complete [10733.847707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10742.628968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10750.642929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10759.274500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10777.053698] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [10786.281393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10786.300336] Lustre: lustre-MDT0000: reset Object Index mappings [10786.749891] LustreError: 260671:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10786.779941] LustreError: 260671:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10786.870613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10791.190300] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10791.915561] LustreError: 260672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10797.031156] LustreError: 260671:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10798.717901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10798.734868] Lustre: lustre-MDT0001: reset Object Index mappings [10799.036829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10803.180584] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10805.825588] Lustre: 261813:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10813.116142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10819.529303] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10823.784883] LustreError: 262166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10828.024776] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10828.269790] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10828.285879] Lustre: Skipped 1 previous similar message [10831.355840] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10831.369162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10834.639738] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10842.395363] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10846.554834] Lustre: 263684:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10852.536731] Lustre: server umount lustre-MDT0000 complete [10855.392780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10855.412823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10855.455536] LustreError: 261405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10857.333817] LustreError: 260652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447442 with bad export cookie 11288942472903297574 [10857.347947] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10857.353339] LustreError: 260652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10857.620527] Lustre: server umount lustre-MDT0001 complete [10872.585892] Lustre: server umount lustre-OST0000 complete [10888.062594] Lustre: server umount lustre-OST0001 complete [10910.759506] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 10:58:13 (1788447493) [10930.605442] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [10941.998375] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [10979.973289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10980.189438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10980.234683] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10980.354152] Lustre: lustre-MDT0000: new disk, initializing [10980.553602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10980.577405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10986.026308] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [10996.986358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10997.108985] Lustre: 267939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10997.163137] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10997.166735] Lustre: Skipped 1 previous similar message [10997.234314] Lustre: lustre-MDT0001: new disk, initializing [10997.377602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10997.385380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11000.970853] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11009.906492] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11015.392446] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11021.958427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11022.255880] Lustre: lustre-OST0000: new disk, initializing [11022.266387] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11022.281351] Lustre: 269908:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11023.945047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11023.952996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11024.068516] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11029.020845] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11038.620620] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11043.264907] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11043.470736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11047.992706] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11048.210212] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11055.078769] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11055.082272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11055.082804] LustreError: Skipped 1 previous similar message [11055.083865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11055.111664] Lustre: Skipped 4 previous similar messages [11056.730534] Lustre: server umount lustre-OST0000 complete [11063.788873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11063.794738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11063.809917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11063.826352] Lustre: Skipped 1 previous similar message [11067.505926] Lustre: server umount lustre-MDT0000 complete [11070.442751] LustreError: 267951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11070.458829] LustreError: 267951:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11072.265222] LustreError: 269907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447657 with bad export cookie 11288942472903299667 [11072.274844] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11072.284481] LustreError: 269907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11072.908550] Lustre: server umount lustre-MDT0001 complete [11094.598257] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [11106.304487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11106.718093] LustreError: 272854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11106.765691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11106.773713] Lustre: Skipped 2 previous similar messages [11111.994448] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11120.798992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11124.678521] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11127.351869] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11133.098117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11138.939528] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11147.497050] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11147.776119] Lustre: lustre-OST0001: new disk, initializing [11147.785615] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11147.797605] Lustre: 275160:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11149.168189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11149.175859] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11149.307508] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11154.718298] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11165.482878] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11169.840612] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11175.693043] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11175.902765] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11187.173989] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11187.180339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11187.196706] Lustre: Skipped 4 previous similar messages [11187.216694] Lustre: Skipped 5 previous similar messages [11187.696829] Lustre: server umount lustre-MDT0000 complete [11191.622600] LustreError: 275171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447776 with bad export cookie 11288942472903300549 [11191.628046] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11191.636127] LustreError: 275171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11191.885925] Lustre: server umount lustre-MDT0001 complete [11205.183672] Lustre: server umount lustre-OST0000 complete [11208.159305] Lustre: 250662:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788447777/real 1788447777] req@ffff9fc975144e00 x1875322739063680/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788447793 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11209.750837] Lustre: server umount lustre-OST0001 complete [11230.369360] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [11241.429644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11241.804957] LustreError: 278506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11241.830195] LustreError: 278506:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11241.884105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11241.888180] Lustre: Skipped 3 previous similar messages [11246.789973] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11256.332198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11260.581750] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11263.627176] Lustre: 279647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11270.162272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11276.303768] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11284.375403] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11286.707061] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11286.707969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11289.756348] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11296.353279] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11299.651642] Lustre: 281523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11304.649843] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11304.694069] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11316.821746] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:04:59 (1788447899) [11333.090229] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11333.102247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11333.115845] Lustre: Skipped 4 previous similar messages [11333.122117] Lustre: Skipped 3 previous similar messages [11337.858388] Lustre: server umount lustre-MDT0000 complete [11338.214654] LustreError: 278501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11338.228743] LustreError: 278501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11341.446791] LustreError: 278488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447926 with bad export cookie 11288942472903307003 [11341.461615] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11343.331209] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11347.844628] Lustre: server umount lustre-MDT0001 complete [11358.563970] Lustre: server umount lustre-OST0000 complete [11363.051621] Lustre: server umount lustre-OST0001 complete [11370.090180] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [11381.184868] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [11433.699016] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [11443.773732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11444.020267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11444.072713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11444.152433] Lustre: lustre-MDT0000: new disk, initializing [11444.267323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11448.997599] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11459.502910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11459.647217] Lustre: 286053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11459.858450] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11464.372659] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11474.761441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11475.047889] Lustre: 287690:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11477.133546] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11481.224932] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11491.480473] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11491.775354] Lustre: 288559:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11493.728451] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11493.748168] Lustre: Skipped 1 previous similar message [11493.827831] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11499.056765] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11507.453161] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11510.691192] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11510.696969] Lustre: Skipped 1 previous similar message [11527.091647] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 11:08:29 (1788448109) [11530.342866] Lustre: server umount lustre-MDT0000 complete [11531.767025] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11531.777271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11531.789311] Lustre: Skipped 2 previous similar messages [11531.801594] LustreError: 286064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11531.817564] LustreError: 286064:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11534.562369] LustreError: 286046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448119 with bad export cookie 11288942472903314941 [11534.569803] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11534.577494] LustreError: 286046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11534.949114] Lustre: server umount lustre-MDT0001 complete [11540.458596] Lustre: server umount lustre-OST0000 complete [11544.127781] Lustre: server umount lustre-OST0001 complete [11553.644563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11554.158696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11554.163247] Lustre: Skipped 7 previous similar messages [11559.098659] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11568.987499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11573.704363] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11586.687764] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11596.714415] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11607.582847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11614.676291] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11622.756500] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11645.501383] loop1: detected capacity change from 0 to 204800000 [11645.684971] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11645.693727] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11645.701981] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11650.423745] loop1: detected capacity change from 0 to 204800000 [11656.581614] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11656.707608] LustreError: 290791:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs12024: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11656.729902] LustreError: 290791:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs12024 [11656.740235] LustreError: 290791:0:(mgs_handler.c:517:mgs_target_reg()) fs12024-MDT0001: Error checking target index: rc = -13 [11656.752358] LustreError: fs12024-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11656.763745] LustreError: 294524:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11656.781796] LustreError: 294524:0:(tgt_mount.c:2132:server_put_super()) no obd fs12024-MDT0001 [11656.793600] LustreError: 294524:0:(tgt_mount.c:116:server_deregister_mount()) fs12024-MDT0001 not registered [11656.838363] Lustre: server umount fs12024-MDT0001 complete [11656.846315] LustreError: 294524:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11667.033645] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11667.086606] Lustre: Setting parameter fs12024-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs12024-MDT0001 [11667.133725] Lustre: srv-fs12024-MDT0001: No data found on store. Initialize space. [11667.136392] Lustre: Skipped 4 previous similar messages [11667.196669] Lustre: fs12024-MDT0001: new disk, initializing [11667.199091] Lustre: Skipped 3 previous similar messages [11667.274204] Lustre: srv-fs12024-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11667.280460] Lustre: 294990:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12024-MDT0001-mdtlov: can't create new llog: rc = -115 [11667.285621] LustreError: 294990:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12024-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11672.528247] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11679.896096] Lustre: server umount fs12024-MDT0001 complete [11690.982786] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11692.697364] loop1: detected capacity change from 0 to 204800000 [11699.165377] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11699.577440] Lustre: 296559:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12024-MDT0001-mdtlov: can't create new llog: rc = -115 [11699.590774] LustreError: 296559:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12024-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11705.362127] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11710.936580] Lustre: server umount fs12024-MDT0001 complete [11721.696908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11721.702919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11721.705992] Lustre: Skipped 6 previous similar messages [11721.718464] Lustre: Skipped 5 previous similar messages [11727.898548] Lustre: server umount lustre-MDT0000 complete [11737.174500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11737.267817] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11742.255815] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11742.712697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11750.469890] loop1: detected capacity change from 0 to 204800000 [11756.656502] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11757.194178] Lustre: 299009:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12024-MDT0001-mdtlov: can't create new llog: rc = -115 [11757.209170] LustreError: 299009:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12024-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11762.535695] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11768.233688] Lustre: server umount fs12024-MDT0001 complete [11788.776828] LustreError: 292635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11788.804872] LustreError: 292635:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 113 previous similar messages [11793.888197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11793.901856] LustreError: Skipped 1 previous similar message [11793.911405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11793.925364] Lustre: Skipped 5 previous similar messages [11796.629431] Lustre: server umount lustre-OST0000 complete [11807.667217] Lustre: server umount lustre-MDT0000 complete [11811.649194] LustreError: 297052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448396 with bad export cookie 11288942472903317622 [11811.669292] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11812.018693] Lustre: server umount lustre-MDT0001 complete [11825.370161] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [11828.841779] Key type lgssc unregistered [11829.145725] LNet: 301353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11829.173829] LNetError: 301353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11829.194715] LNet: Removed LNI 192.168.204.108@tcp [11830.331167] Key type .llcrypt unregistered [11830.334270] Key type ._llcrypt unregistered [11850.326901] Key type ._llcrypt registered [11850.331492] Key type .llcrypt registered [11850.478524] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [11851.482475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11851.539088] alg: No test for adler32 (adler32-zlib) [11852.742342] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11852.970640] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [11854.623175] Key type lgssc registered [11855.938263] Lustre: Echo OBD driver; http://www.lustre.org/ [11864.569926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11864.602529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11866.057573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11870.978503] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11875.302304] LustreError: 302635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11875.327844] LustreError: 302635:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11880.043573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11880.352822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11884.800708] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11892.935503] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11899.879884] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11906.146299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11906.494192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11910.570642] LustreError: 304463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11911.911130] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11915.748479] LustreError: 304720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11919.796826] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11920.869649] LustreError: 304463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11926.022308] LustreError: 304463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11926.039356] LustreError: 304463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11936.245123] LustreError: 304720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11936.282247] LustreError: 304720:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11946.212077] loop1: detected capacity change from 0 to 204800000 [11946.777175] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11946.814371] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11946.878232] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11953.021897] loop1: detected capacity change from 0 to 204800000 [11953.472136] LustreError: 304464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11953.502875] LustreError: 304464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11959.634684] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11959.777555] Lustre: fs29515-OST0000: new disk, initializing [11959.781342] Lustre: srv-fs29515-OST0000: No data found on store. Initialize space. [11959.789699] Lustre: 306370:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11959.880774] Lustre: fs29515-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11968.239764] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [11978.439680] Lustre: server umount fs29515-OST0000 complete [11987.442749] LustreError: 304465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11987.462989] LustreError: 304465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [11992.584824] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11994.199480] loop1: detected capacity change from 0 to 204800000 [12002.116411] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12002.356846] Lustre: fs29515-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12007.911386] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12010.173714] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12021.097137] loop2: detected capacity change from 0 to 204800000 [12021.313418] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12021.339295] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12021.356470] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12027.364827] loop2: detected capacity change from 0 to 204800000 [12034.321566] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12034.389719] LustreError: fs29515-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12034.396626] LustreError: fs29515-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12034.405209] LustreError: 309416:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [12034.413324] LustreError: 309416:0:(tgt_mount.c:2132:server_put_super()) no obd fs29515-OST0001 [12034.422395] LustreError: 309416:0:(tgt_mount.c:116:server_deregister_mount()) fs29515-OST0001 not registered [12034.461920] Lustre: server umount fs29515-OST0001 complete [12034.465155] LustreError: 309416:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12046.756126] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12047.059931] Lustre: fs29515-OST0001: new disk, initializing [12047.070822] Lustre: srv-fs29515-OST0001: No data found on store. Initialize space. [12047.078398] Lustre: 309968:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12047.198546] Lustre: fs29515-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12053.999674] LustreError: 304463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12054.029589] LustreError: 304463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [12054.870866] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12063.774771] Lustre: server umount fs29515-OST0000 complete [12074.154822] Lustre: server umount fs29515-OST0001 complete [12087.239454] loop1: detected capacity change from 0 to 204800000 [12093.127373] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12093.533635] Lustre: fs29515-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12100.832514] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12109.170144] loop2: detected capacity change from 0 to 204800000 [12115.512615] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12122.259906] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12131.958885] Lustre: server umount fs29515-OST0000 complete [12139.766134] Lustre: server umount fs29515-OST0001 complete [12146.159678] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12156.389914] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12156.392577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12156.393551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12156.400361] LustreError: Skipped 1 previous similar message [12156.416406] Lustre: Skipped 1 previous similar message [12161.518731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12161.524025] Lustre: Skipped 1 previous similar message [12161.531913] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12166.112459] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12166.253782] Lustre: server umount lustre-OST0000 complete [12171.752500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12171.764830] Lustre: Skipped 1 previous similar message [12171.765198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12171.779175] Lustre: Skipped 2 previous similar messages [12176.871034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12176.878434] Lustre: Skipped 1 previous similar message [12177.390584] Lustre: server umount lustre-MDT0000 complete [12181.462579] LustreError: 302616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448766 with bad export cookie 3348191522307442368 [12181.478187] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12181.488726] LustreError: 302616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12181.866968] Lustre: server umount lustre-MDT0001 complete [12194.061112] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [12196.765172] Key type lgssc unregistered [12197.006668] LNet: 315893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12197.012586] LNetError: 315893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12197.035742] LNet: Removed LNI 192.168.204.108@tcp [12197.987176] Key type .llcrypt unregistered [12197.990122] Key type ._llcrypt unregistered [12232.738791] Key type ._llcrypt registered [12232.741836] Key type .llcrypt registered [12232.904713] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [12251.285668] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [12252.330655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12252.455932] alg: No test for adler32 (adler32-zlib) [12253.477766] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12253.750063] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [12255.463205] Key type lgssc registered [12256.690786] Lustre: Echo OBD driver; http://www.lustre.org/ [12296.336877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12296.395805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12297.726743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12297.746115] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12297.823775] Lustre: lustre-MDT0000: new disk, initializing [12297.932756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12297.948623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12302.964511] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12314.535958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12314.673827] Lustre: 319753:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12314.790766] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12314.798736] Lustre: Skipped 1 previous similar message [12314.866846] Lustre: lustre-MDT0001: new disk, initializing [12314.965420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12315.037727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12315.050505] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12322.550393] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12336.576370] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12344.603581] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12350.879376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12351.196220] Lustre: lustre-OST0000: new disk, initializing [12351.199977] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12351.204254] Lustre: 321722:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12351.248507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12356.716797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12356.728487] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12356.855451] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12357.679426] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12368.974864] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12371.761165] Lustre: Mounted lustre-client - version 2.17.58_2_gee80f9b [12373.062517] Lustre: Unmounted lustre-client [12387.300438] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12387.301231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12387.318871] Lustre: Skipped 1 previous similar message [12387.325527] Lustre: Skipped 1 previous similar message [12388.966278] Lustre: server umount lustre-OST0000 complete [12397.036087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12397.042735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12397.053922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12399.413648] Lustre: server umount lustre-MDT0000 complete [12403.354362] LustreError: 319745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448988 with bad export cookie 16218697697552111506 [12403.358428] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12403.360547] LustreError: 319745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12403.755988] Lustre: server umount lustre-MDT0001 complete [12416.340951] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [12419.921479] Key type lgssc unregistered [12422.753718] LNet: 1 local NIs in recovery (showing 1): 192.168.204.108@tcp [12423.393733] LNet: 323885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12423.400541] LNetError: 323885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12502.252581] LNet: Removed LNI 192.168.204.108@tcp [12503.806244] Key type .llcrypt unregistered [12503.810060] Key type ._llcrypt unregistered [12527.798612] Key type ._llcrypt registered [12527.800615] Key type .llcrypt registered [12527.914320] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [12546.216393] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [12547.878875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12547.927047] alg: No test for adler32 (adler32-zlib) [12549.135508] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12549.580176] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [12551.327180] Key type lgssc registered [12552.634735] Lustre: Echo OBD driver; http://www.lustre.org/ [12588.089960] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 11:26:10 (1788449170) [12596.463736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12596.501063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12597.915419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12597.952532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12598.017314] Lustre: lustre-MDT0000: new disk, initializing [12598.084728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12598.114667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12603.137509] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12614.938235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12615.077634] Lustre: 327105:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12615.157283] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12615.164901] Lustre: Skipped 1 previous similar message [12615.324490] Lustre: lustre-MDT0001: new disk, initializing [12615.455184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12615.541619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12615.565554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12620.858182] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12631.953580] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12638.430307] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12646.071607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12646.395137] Lustre: lustre-OST0000: new disk, initializing [12646.400065] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12646.407229] Lustre: 329075:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12646.559367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12653.763311] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12655.703669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12655.715161] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12655.817459] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12663.699441] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12681.696515] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12681.702539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12681.704203] LustreError: Skipped 1 previous similar message [12681.717769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12681.724374] Lustre: Skipped 1 previous similar message [12684.863826] Lustre: server umount lustre-OST0000 complete [12691.938340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12691.941639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12691.963204] Lustre: Skipped 2 previous similar messages [12691.974287] Lustre: Skipped 1 previous similar message [12695.827820] Lustre: server umount lustre-MDT0000 complete [12699.687339] LustreError: 329074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449284 with bad export cookie 17160283365304291485 [12699.689732] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12699.698569] LustreError: 329074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12700.023676] Lustre: server umount lustre-MDT0001 complete [12713.920396] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [12717.534633] Key type lgssc unregistered [12717.885141] LNet: 330985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12717.897582] LNetError: 330985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12717.927748] LNet: Removed LNI 192.168.204.108@tcp [12718.897780] Key type .llcrypt unregistered [12718.900468] Key type ._llcrypt unregistered [12744.307968] Key type ._llcrypt registered [12744.316128] Key type .llcrypt registered [12744.410706] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [12746.049670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12746.102181] alg: No test for adler32 (adler32-zlib) [12747.172511] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12747.456397] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [12749.191163] Key type lgssc registered [12750.584540] Lustre: Echo OBD driver; http://www.lustre.org/ [12759.231250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12759.269197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12760.711712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12765.100367] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12770.278809] LustreError: 332258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12770.312057] LustreError: 332258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12774.148537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12774.687170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12780.158357] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12790.100234] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12798.408974] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12806.045478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12806.561498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12813.644992] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12821.824796] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12842.447862] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12844.095638] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12849.520483] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.8@tcp (stopping) [12850.664068] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12850.675815] Lustre: Skipped 1 previous similar message [12850.676602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12850.691074] Lustre: Skipped 2 previous similar messages [12854.480379] Lustre: server umount lustre-MDT0000 complete [12854.603220] LustreError: 332252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12855.778705] LustreError: 333419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12855.793969] LustreError: 333419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12858.201800] LustreError: 332238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449443 with bad export cookie 7350975285358742738 [12858.204276] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12858.220393] LustreError: 332238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12859.725221] LustreError: 332254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12859.760728] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.8@tcp (stopping) [12860.906822] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12863.968372] LustreError: 332254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12863.968824] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12863.989538] LustreError: 332254:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12864.002981] Lustre: Skipped 1 previous similar message [12864.802177] Lustre: server umount lustre-MDT0001 complete [12869.037605] Lustre: server umount lustre-OST0000 complete [12878.130443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12882.816620] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12885.924836] Lustre: 336462:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.108@tcp [12888.554942] Lustre: server umount MGS complete [12898.093356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12898.543325] LustreError: 337034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12898.674971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12903.457553] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12912.674762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12913.390860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12918.878582] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12929.322075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12929.608413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12937.326616] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [12945.050139] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12946.634876] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12951.713059] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12953.442715] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12959.208101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12959.209581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12959.215664] LustreError: Skipped 1 previous similar message [12959.218702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12959.239918] Lustre: Skipped 1 previous similar message [12960.500888] Lustre: server umount lustre-OST0000 complete [12969.444995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12969.445835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12969.458714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12969.480489] Lustre: Skipped 1 previous similar message [12969.491227] Lustre: Skipped 1 previous similar message [12970.787920] Lustre: server umount lustre-MDT0000 complete [12974.564726] LustreError: 339090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12974.586248] LustreError: 339090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [12974.677945] LustreError: 337016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449559 with bad export cookie 7350975285358743578 [12974.689048] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12974.695535] LustreError: 337016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12975.306697] Lustre: server umount lustre-MDT0001 complete [12988.961307] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [12991.906431] Key type lgssc unregistered [12992.294714] LNet: 340469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12992.309686] LNetError: 340469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12992.330711] LNet: Removed LNI 192.168.204.108@tcp [12993.282398] Key type .llcrypt unregistered [12993.285814] Key type ._llcrypt unregistered [13012.820459] Key type ._llcrypt registered [13012.822650] Key type .llcrypt registered [13012.901919] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [13030.642468] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [13031.618599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13031.689359] alg: No test for adler32 (adler32-zlib) [13032.734048] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13033.053830] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [13034.720080] Key type lgssc registered [13036.085655] Lustre: Echo OBD driver; http://www.lustre.org/ [13078.254171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13078.274299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13079.480625] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13079.552613] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13079.725300] Lustre: lustre-MDT0000: new disk, initializing [13079.871825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13079.890518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13085.505741] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13104.147047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13104.291119] Lustre: 343627:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13104.354411] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13104.360523] Lustre: Skipped 1 previous similar message [13104.433178] Lustre: lustre-MDT0001: new disk, initializing [13104.547435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13104.594677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13104.630603] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13110.790406] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13120.161516] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13127.143174] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13135.249604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13135.625762] Lustre: lustre-OST0000: new disk, initializing [13135.629694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13135.637147] Lustre: 345599:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13135.722868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13141.620851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13141.644824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13141.808560] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13142.369804] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13152.716742] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13157.307636] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13157.506034] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13161.556728] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13161.744547] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13167.595349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13167.595633] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13167.608306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13167.611328] Lustre: Skipped 1 previous similar message [13167.625535] LustreError: Skipped 1 previous similar message [13172.713362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13172.723297] Lustre: Skipped 1 previous similar message [13177.827960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13177.844876] Lustre: Skipped 2 previous similar messages [13178.335166] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13178.551177] Lustre: server umount lustre-OST0000 complete [13186.530096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13186.546097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13186.576950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13188.068663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13189.817802] Lustre: server umount lustre-MDT0000 complete [13193.923363] LustreError: 343622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449778 with bad export cookie 725473921248082384 [13193.927607] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13193.930499] LustreError: 343622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13194.319916] Lustre: server umount lustre-MDT0001 complete [13204.287156] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 11:36:27 (1788449787) [13223.233712] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [13236.070229] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [13273.705445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13273.980908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13274.039234] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13274.153074] Lustre: lustre-MDT0000: new disk, initializing [13274.269348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13274.301548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13278.868409] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13289.068543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13289.182841] Lustre: 350362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13289.230820] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13289.235527] Lustre: Skipped 1 previous similar message [13289.308593] Lustre: lustre-MDT0001: new disk, initializing [13289.398398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13289.433076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13289.444252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13293.439934] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13303.369521] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13309.352703] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13316.113459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13316.430311] Lustre: lustre-OST0000: new disk, initializing [13316.434083] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13316.441532] Lustre: 352328:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13316.525714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13322.876932] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13323.843553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13323.859523] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13323.983911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13332.280179] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13365.218830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13365.218971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13365.227176] LustreError: Skipped 1 previous similar message [13365.230143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13365.235082] Lustre: Skipped 1 previous similar message [13365.240258] Lustre: Skipped 2 previous similar messages [13375.460142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13375.466096] Lustre: Skipped 3 previous similar messages [13378.015126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13378.185754] Lustre: server umount lustre-OST0000 complete [13385.703399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13385.734716] Lustre: Skipped 1 previous similar message [13388.638861] Lustre: server umount lustre-MDT0000 complete [13390.820300] LustreError: 350373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13390.839842] LustreError: 350373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13392.094061] LustreError: 350352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449977 with bad export cookie 725473921248083266 [13392.097730] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13392.112652] LustreError: 350352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [13392.455390] Lustre: server umount lustre-MDT0001 complete [13402.915430] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [13405.310481] Key type lgssc unregistered [13405.552688] LNet: 354501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13405.563716] LNetError: 354501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13406.629610] LNet: Removed LNI 192.168.204.108@tcp [13407.629543] Key type .llcrypt unregistered [13407.631677] Key type ._llcrypt unregistered [13426.582281] Key type ._llcrypt registered [13426.586420] Key type .llcrypt registered [13426.719888] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [13427.636835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13427.727713] alg: No test for adler32 (adler32-zlib) [13428.816721] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13429.044295] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [13430.752483] Key type lgssc registered [13432.510515] Lustre: Echo OBD driver; http://www.lustre.org/ [13441.757202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13441.770986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13443.155886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13447.602463] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13453.280537] LustreError: 355772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13453.304552] LustreError: 355772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13457.293428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13457.656321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13463.760603] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13472.655132] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13478.955295] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13486.788826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13487.138209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13493.481993] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13500.961462] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13513.697795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13513.704629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13513.722461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13518.305952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13518.312726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13518.317132] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13518.336565] Lustre: Skipped 1 previous similar message [13519.772669] Lustre: server umount lustre-OST0000 complete [13524.449214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13524.458925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13524.474420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13528.554995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13528.556505] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13530.761313] Lustre: server umount lustre-MDT0000 complete [13533.676929] LustreError: 355773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13533.701099] LustreError: 355773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13535.532914] LustreError: 355755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450120 with bad export cookie 432967592427416994 [13535.533982] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13535.558459] LustreError: 355755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13536.191775] Lustre: server umount lustre-MDT0001 complete [13549.790524] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [13553.318570] Key type lgssc unregistered [13553.870821] LNet: 361307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13553.888792] LNetError: 361307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13554.938551] LNet: Removed LNI 192.168.204.108@tcp [13556.305961] Key type .llcrypt unregistered [13556.315343] Key type ._llcrypt unregistered [13576.813130] Key type ._llcrypt registered [13576.814315] Key type .llcrypt registered [13576.934870] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [13577.941690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13577.990108] alg: No test for adler32 (adler32-zlib) [13579.140834] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13579.451374] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [13581.151273] Key type lgssc registered [13582.933320] Lustre: Echo OBD driver; http://www.lustre.org/ [13593.172825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13593.218302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13595.064059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13601.129357] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13604.328145] LustreError: 365025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13604.353859] LustreError: 365025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13609.440038] LustreError: 365027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13612.362260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13612.833044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13618.568309] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13628.300217] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13636.526226] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13645.425431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13645.884724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13654.661246] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13664.241646] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13674.466993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13674.475603] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13674.496621] Lustre: Skipped 1 previous similar message [13674.499626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13674.508925] Lustre: Skipped 1 previous similar message [13678.923895] Lustre: server umount lustre-OST0000 complete [13684.706244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13684.710066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13684.716383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13684.747895] Lustre: Skipped 1 previous similar message [13689.827662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13689.831644] Lustre: Skipped 1 previous similar message [13690.677753] Lustre: server umount lustre-MDT0000 complete [13694.973223] LustreError: 365027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13695.006214] LustreError: 365027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13695.982373] LustreError: 364961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450280 with bad export cookie 2764308835963566906 [13695.991479] LustreError: 364961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13695.996547] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13696.616275] Lustre: server umount lustre-MDT0001 complete [13711.997626] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [13716.188901] Key type lgssc unregistered [13716.705634] LNet: 372539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13716.720681] LNetError: 372539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13716.748373] LNet: Removed LNI 192.168.204.108@tcp [13718.246636] Key type .llcrypt unregistered [13718.247913] Key type ._llcrypt unregistered [13739.209316] Key type ._llcrypt registered [13739.213088] Key type .llcrypt registered [13739.408910] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [13741.084749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13741.111666] alg: No test for adler32 (adler32-zlib) [13742.286754] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13742.606223] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [13744.328215] Key type lgssc registered [13745.592259] Lustre: Echo OBD driver; http://www.lustre.org/ [13754.760980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13754.816045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13756.382384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13761.284984] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13766.633610] LustreError: 376415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13766.647436] LustreError: 376415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13770.063159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13770.489372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13775.744622] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13785.558097] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13793.187788] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13801.358575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13801.947892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13810.455434] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [13819.791630] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13840.867950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13840.872561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13840.880801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13841.896284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13841.896827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13841.922744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13845.797146] Lustre: server umount lustre-OST0000 complete [13852.129343] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13852.134620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13852.140577] Lustre: Skipped 1 previous similar message [13852.160863] Lustre: Skipped 2 previous similar messages [13855.645200] Lustre: server umount lustre-MDT0000 complete [13857.255197] LustreError: 376413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13857.295950] LustreError: 376413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13859.280254] LustreError: 376338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450444 with bad export cookie 6004110287398598067 [13859.288503] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13859.290888] LustreError: 376338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13859.809083] Lustre: server umount lustre-MDT0001 complete [13870.545982] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [13873.429775] Key type lgssc unregistered [13873.763951] LNet: 382582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13873.774877] LNetError: 382582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13874.793170] LNet: Removed LNI 192.168.204.108@tcp [13875.656565] Key type .llcrypt unregistered [13875.660132] Key type ._llcrypt unregistered [13984.213936] Key type ._llcrypt registered [13984.218353] Key type .llcrypt registered [13984.315780] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [13984.583178] Key type .llcrypt unregistered [13984.589614] Key type ._llcrypt unregistered [14066.652294] Key type ._llcrypt registered [14066.660149] Key type .llcrypt registered [14066.802263] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14067.180475] Key type .llcrypt unregistered [14067.183265] Key type ._llcrypt unregistered [14100.720744] Key type ._llcrypt registered [14100.722853] Key type .llcrypt registered [14100.839748] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14101.069348] Key type .llcrypt unregistered [14101.072201] Key type ._llcrypt unregistered [14135.407305] Key type ._llcrypt registered [14135.410442] Key type .llcrypt registered [14135.535754] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14136.047426] Key type .llcrypt unregistered [14136.051434] Key type ._llcrypt unregistered [14218.676051] Key type ._llcrypt registered [14218.677887] Key type .llcrypt registered [14218.805433] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14219.027501] Key type .llcrypt unregistered [14219.030885] Key type ._llcrypt unregistered [14244.234400] Key type ._llcrypt registered [14244.237884] Key type .llcrypt registered [14244.370595] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14244.681962] Key type .llcrypt unregistered [14244.687096] Key type ._llcrypt unregistered [14277.031585] Key type ._llcrypt registered [14277.033355] Key type .llcrypt registered [14277.188654] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14277.542325] Key type .llcrypt unregistered [14277.544956] Key type ._llcrypt unregistered [14310.470536] Key type ._llcrypt registered [14310.471894] Key type .llcrypt registered [14310.568543] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14310.877291] Key type .llcrypt unregistered [14310.879889] Key type ._llcrypt unregistered [14343.640792] Key type ._llcrypt registered [14343.642977] Key type .llcrypt registered [14343.768697] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14344.024623] Key type .llcrypt unregistered [14344.030521] Key type ._llcrypt unregistered [14373.720644] Key type ._llcrypt registered [14373.723451] Key type .llcrypt registered [14373.811856] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14374.127301] Key type .llcrypt unregistered [14374.129137] Key type ._llcrypt unregistered [14447.106813] Key type ._llcrypt registered [14447.111823] Key type .llcrypt registered [14447.263377] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14447.654354] Key type .llcrypt unregistered [14447.660208] Key type ._llcrypt unregistered [14483.234468] Key type ._llcrypt registered [14483.240724] Key type .llcrypt registered [14483.378142] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14483.689163] Key type .llcrypt unregistered [14483.696355] Key type ._llcrypt unregistered [14517.869701] Key type ._llcrypt registered [14517.873323] Key type .llcrypt registered [14517.996468] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14518.365315] Key type .llcrypt unregistered [14518.368980] Key type ._llcrypt unregistered [14573.330933] Key type ._llcrypt registered [14573.335431] Key type .llcrypt registered [14573.449489] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [14590.037529] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [14591.910668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14591.935296] alg: No test for adler32 (adler32-zlib) [14593.058981] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14593.285133] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [14594.920544] Key type lgssc registered [14596.050225] Lustre: Echo OBD driver; http://www.lustre.org/ [14637.270038] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14637.294888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14638.568746] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14638.630500] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14638.749619] Lustre: tst250fs-MDT0000: new disk, initializing [14638.892116] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14638.918993] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14644.865433] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [14656.669665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14656.883740] Lustre: 403097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14656.972158] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14656.983313] Lustre: Skipped 1 previous similar message [14657.104600] Lustre: tst250fs-MDT0001: new disk, initializing [14657.234851] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14657.301150] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14657.317933] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14663.063941] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [14668.185626] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14674.544386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14674.774199] Lustre: tst250fs-OST0000: new disk, initializing [14674.782563] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14674.789777] Lustre: 404727:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14674.911990] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14679.047446] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14679.056909] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14679.196978] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14682.564278] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [14693.943707] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14694.166953] Lustre: tst250fs-OST0001: new disk, initializing [14694.172213] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14694.192949] Lustre: 405609:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14694.318659] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14701.321125] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all [14703.207501] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14703.214397] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14703.295128] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14708.327552] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14710.757227] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14710.772630] Lustre: Skipped 1 previous similar message [14734.324895] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14734.332142] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14734.341889] Lustre: Skipped 1 previous similar message [14734.343602] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14734.347855] Lustre: Skipped 1 previous similar message [14738.332375] Lustre: server umount tst250fs-MDT0000 complete [14742.590755] LustreError: 403088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451327 with bad export cookie 15890903820511217923 [14742.602028] LustreError: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14742.611568] LustreError: 403088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14743.081197] Lustre: server umount tst250fs-MDT0001 complete [14757.870336] Lustre: server umount tst250fs-OST0000 complete [14760.799503] Lustre: 401316:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788451329/real 1788451329] req@ffff9fc977189c00 x1875326982495744/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788451345 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14760.820607] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14760.841567] Lustre: Skipped 2 previous similar messages [14762.747410] Lustre: server umount tst250fs-OST0001 complete [14784.789983] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [14794.661688] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [14832.500109] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:03:34 (1788451414) [14834.760509] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14857.885909] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [14868.229420] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [14906.707948] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14567 sec ========= 12:04:49 (1788451489) [14909.050952] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:04:51 (1788451491) === [14913.340460] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:04:56 (1788451496) === [14960.156441] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [14963.014354] Key type lgssc unregistered [14963.301893] LNet: 414764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14963.326068] LNetError: 414764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14963.345859] LNet: Removed LNI 192.168.204.108@tcp [14964.028522] Key type .llcrypt unregistered [14964.030749] Key type ._llcrypt unregistered