[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 472429892 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2544MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001019] APIC: Switch to symmetric I/O mode setup [ 0.002412] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005023] kvm-guest: setup PV IPIs [ 0.008692] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009033] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010020] pid_max: default: 32768 minimum: 301 [ 0.011172] LSM: Security Framework initializing [ 0.012079] Yama: becoming mindful. [ 0.013074] SELinux: Initializing. [ 0.014112] *** VALIDATE selinux *** [ 0.024108] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029737] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030243] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031142] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033179] *** VALIDATE tmpfs *** [ 0.035251] *** VALIDATE proc *** [ 0.037082] *** VALIDATE cgroup *** [ 0.038018] *** VALIDATE cgroup2 *** [ 0.039329] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040195] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043045] Spectre V2 : User space: Vulnerable [ 0.044020] Speculative Store Bypass: Vulnerable [ 0.047127] debug: unmapping init [mem 0xfffffffface59000-0xfffffffface60fff] [ 0.050214] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051807] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052035] ... version: 2 [ 0.053023] ... bit width: 48 [ 0.054022] ... generic registers: 4 [ 0.055019] ... value mask: 0000ffffffffffff [ 0.056023] ... max period: 00007fffffffffff [ 0.057025] ... fixed-purpose events: 3 [ 0.058023] ... event mask: 000000070000000f [ 0.060312] rcu: Hierarchical SRCU implementation. [ 0.062858] smp: Bringing up secondary CPUs ... [ 0.063777] x86: Booting SMP configuration: [ 0.064031] .... node #0, CPUs: #1 #2 #3 [ 0.068106] smp: Brought up 1 node, 4 CPUs [ 0.070014] smpboot: Max logical packages: 1 [ 0.071014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.154036] node 0 deferred pages initialised in 81ms [ 0.159157] devtmpfs: initialized [ 0.160285] x86/mm: Memory block size: 128MB [ 0.162891] gcov: version magic: 0x41383552 [ 0.164418] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.167075] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.168316] pinctrl core: initialized pinctrl subsystem [ 0.170193] [ 0.170566] ************************************************************* [ 0.173016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174009] ** ** [ 0.176028] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.177011] ** ** [ 0.179014] ** This means that this kernel is built to expose internal ** [ 0.181014] ** IOMMU data structures, which may compromise security on ** [ 0.182015] ** your system. ** [ 0.184010] ** ** [ 0.186016] ** If you see this message and you are not debugging the ** [ 0.187010] ** kernel, report this immediately to your vendor! ** [ 0.189014] ** ** [ 0.191013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.192009] ************************************************************* [ 0.194728] NET: Registered protocol family 16 [ 0.196486] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.198105] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.199054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.203025] cpuidle: using governor menu [ 0.205028] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.208071] PCI: Using configuration type 1 for base access [ 0.210147] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.218104] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.220030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.223126] cryptd: max_cpu_qlen set to 1000 [ 0.226296] ACPI: Added _OSI(Module Device) [ 0.227017] ACPI: Added _OSI(Processor Device) [ 0.228011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.230012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.235952] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.241324] ACPI: Interpreter enabled [ 0.244069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.246014] ACPI: Using IOAPIC for interrupt routing [ 0.248171] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.253774] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.267475] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.270074] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.273119] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.277094] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.282781] acpiphp: Slot [2] registered [ 0.283111] acpiphp: Slot [5] registered [ 0.285136] acpiphp: Slot [6] registered [ 0.286108] acpiphp: Slot [7] registered [ 0.288178] acpiphp: Slot [8] registered [ 0.291157] acpiphp: Slot [9] registered [ 0.292130] acpiphp: Slot [10] registered [ 0.294137] acpiphp: Slot [3] registered [ 0.296108] acpiphp: Slot [4] registered [ 0.297238] acpiphp: Slot [11] registered [ 0.299114] acpiphp: Slot [12] registered [ 0.300208] acpiphp: Slot [13] registered [ 0.302128] acpiphp: Slot [14] registered [ 0.303133] acpiphp: Slot [15] registered [ 0.305133] acpiphp: Slot [16] registered [ 0.307135] acpiphp: Slot [17] registered [ 0.308130] acpiphp: Slot [18] registered [ 0.310112] acpiphp: Slot [19] registered [ 0.313145] acpiphp: Slot [20] registered [ 0.315097] acpiphp: Slot [21] registered [ 0.316192] acpiphp: Slot [22] registered [ 0.318142] acpiphp: Slot [23] registered [ 0.320155] acpiphp: Slot [24] registered [ 0.321142] acpiphp: Slot [25] registered [ 0.323116] acpiphp: Slot [26] registered [ 0.325147] acpiphp: Slot [27] registered [ 0.327122] acpiphp: Slot [28] registered [ 0.328130] acpiphp: Slot [29] registered [ 0.330145] acpiphp: Slot [30] registered [ 0.332126] acpiphp: Slot [31] registered [ 0.333064] PCI host bridge to bus 0000:00 [ 0.335025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.337031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.340028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.343035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.346037] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.349032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.351200] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.354338] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.358695] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.370838] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.374500] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.378057] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.380022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.384024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.387624] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.389881] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.393551] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.396810] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.403028] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.415990] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.421017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.429622] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.441019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.449021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.468017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.476678] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.486017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.495017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.519016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.529321] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.541028] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.553020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.574022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.586582] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.593020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.605027] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.632038] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.643444] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.651017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.659019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.680020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.692679] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.702020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.713142] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.735017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.747421] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.750465] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.752384] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.755391] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.757240] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.764131] iommu: Default domain type: Passthrough [ 0.766716] SCSI subsystem initialized [ 0.769212] ACPI: bus type USB registered [ 0.771115] usbcore: registered new interface driver usbfs [ 0.774107] usbcore: registered new interface driver hub [ 0.776093] usbcore: registered new device driver usb [ 0.778260] pps_core: LinuxPPS API ver. 1 registered [ 0.780013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.783073] PTP clock support registered [ 0.785060] EDAC MC: Ver: 3.0.0 [ 0.787547] PCI: Using ACPI for IRQ routing [ 0.789988] NetLabel: Initializing [ 0.791022] NetLabel: domain hash size = 128 [ 0.792009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.794149] NetLabel: unlabeled traffic allowed by default [ 0.797122] vgaarb: loaded [ 0.798185] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.800011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.806455] clocksource: Switched to clocksource kvm-clock [ 0.937484] VFS: Disk quotas dquot_6.6.0 [ 0.941991] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.945429] *** VALIDATE ramfs *** [ 0.947269] *** VALIDATE hugetlbfs *** [ 0.949272] pnp: PnP ACPI init [ 0.952492] pnp: PnP ACPI: found 6 devices [ 0.973435] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.977627] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.979840] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.982473] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.985163] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.987518] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.990393] NET: Registered protocol family 2 [ 0.992932] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.997800] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.002119] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.007964] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.012303] TCP: Hash tables configured (established 65536 bind 65536) [ 1.015802] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.019414] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.022665] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.026213] NET: Registered protocol family 1 [ 1.029547] RPC: Registered named UNIX socket transport module. [ 1.031763] RPC: Registered udp transport module. [ 1.033674] RPC: Registered tcp transport module. [ 1.035623] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.038221] NET: Registered protocol family 44 [ 1.040242] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.042975] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.045359] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.048101] PCI: CLS 0 bytes, default 64 [ 1.051272] Unpacking initramfs... [ 2.487255] debug: unmapping init [mem 0xffff94943cc54000-0xffff94943ffbffff] [ 2.491881] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.496974] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.501360] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.041255] Initialise system trusted keyrings [ 3.045519] Key type blacklist registered [ 3.047560] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.057358] zbud: loaded [ 3.060637] *** VALIDATE nfs *** [ 3.061925] *** VALIDATE nfs4 *** [ 3.063679] pstore: using deflate compression [ 3.067403] Platform Keyring initialized [ 3.181974] NET: Registered protocol family 38 [ 3.183382] Key type asymmetric registered [ 3.184321] Asymmetric key parser 'x509' registered [ 3.186215] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.189801] io scheduler mq-deadline registered [ 3.192627] io scheduler kyber registered [ 3.195402] io scheduler bfq registered [ 3.198426] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.201740] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.204723] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.207692] ACPI: Power Button [PWRF] [ 3.213693] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.220930] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.238888] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.247986] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.270080] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.301666] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.329661] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.335202] Non-volatile memory driver v1.3 [ 3.336927] Linux agpgart interface v0.103 [ 3.371635] virtio_blk virtio1: [vda] 146720 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.375128] vda: detected capacity change from 0 to 75120640 [ 3.393502] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.397283] vdb: detected capacity change from 0 to 1073741824 [ 3.414747] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.417885] vdc: detected capacity change from 0 to 2621440000 [ 3.450513] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.454313] vdd: detected capacity change from 0 to 2621440000 [ 3.478692] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.482074] vde: detected capacity change from 0 to 4294967296 [ 3.499345] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.502683] vdf: detected capacity change from 0 to 4294967296 [ 3.511691] libphy: Fixed MDIO Bus: probed [ 3.517376] usbcore: registered new interface driver usbserial_generic [ 3.520127] usbserial: USB Serial support registered for generic [ 3.524627] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.530673] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.533392] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.536461] mousedev: PS/2 mouse device common for all mice [ 3.539967] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.545068] rtc_cmos 00:05: RTC can wake from S4 [ 3.549708] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.551062] rtc_cmos 00:05: registered as rtc0 [ 3.559167] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.562822] intel_pstate: CPU model not supported [ 3.565995] hid: raw HID events driver (C) Jiri Kosina [ 3.568432] usbcore: registered new interface driver usbhid [ 3.569226] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.574158] usbhid: USB HID core driver [ 3.574506] drop_monitor: Initializing network drop monitor service [ 3.583110] Initializing XFRM netlink socket [ 3.585446] NET: Registered protocol family 10 [ 3.589729] Segment Routing with IPv6 [ 3.591536] NET: Registered protocol family 17 [ 3.594204] mpls_gso: MPLS GSO support [ 3.599663] RAS: Correctable Errors collector initialized. [ 3.602057] AVX version of gcm_enc/dec engaged. [ 3.603839] AES CTR mode by8 optimization enabled [ 3.692599] sched_clock: Marking stable (3692549548, 0)->(4570500234, -877950686) [ 3.697328] registered taskstats version 1 [ 3.699275] Loading compiled-in X.509 certificates [ 3.701673] zswap: loaded using pool lzo/zbud [ 3.730145] Key type big_key registered [ 3.745063] Key type encrypted registered [ 3.746765] ima: No TPM chip found, activating TPM-bypass! [ 3.749298] ima: Allocated hash algorithm: sha1 [ 3.751388] ima: No architecture policies found [ 3.753166] evm: Initialising EVM extended attributes: [ 3.755394] evm: security.selinux [ 3.756605] evm: security.ima [ 3.757772] evm: security.capability [ 3.759189] evm: HMAC attrs: 0x1 [ 3.761807] rtc_cmos 00:05: setting system clock to 2026-09-08 04:50:47 UTC (1788843047) [ 3.768815] debug: unmapping init [mem 0xffffffffade03000-0xffffffffadffffff] [ 3.772373] debug: unmapping init [mem 0xffffffffacb82000-0xfffffffface58fff] [ 3.781094] Write protecting the kernel read-only data: 28672k [ 3.785696] debug: unmapping init [mem 0xffffffffab203000-0xffffffffab3fffff] [ 3.788530] debug: unmapping init [mem 0xffffffffabb14000-0xffffffffabbfffff] [ 3.823845] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.832624] systemd[1]: Detected virtualization kvm. [ 3.835237] systemd[1]: Detected architecture x86-64. [ 3.837947] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.866205] systemd[1]: No hostname configured. [ 3.868448] systemd[1]: Set hostname to . [ 3.870858] random: systemd: uninitialized urandom read (16 bytes read) [ 3.873741] systemd[1]: Initializing machine ID from random generator. [ 3.922821] random: ln: uninitialized urandom read (6 bytes read) [ 4.052373] random: systemd: uninitialized urandom read (16 bytes read) [ 4.056090] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.061155] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.064454] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.697739] device-mapper: uevent: version 1.0.3 [ 4.700509] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.420929] virtio_net virtio0 ens2: renamed from eth0 [ 5.447483] random: fast init done [ 5.516490] scsi host0: ata_piix [ 5.568413] scsi host1: ata_piix [ 5.570109] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.573050] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.294135] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.274897] random: crng init done [ 10.277092] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.639490] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.642891] printk: systemd: 25 output lines suppressed due to ratelimiting [ 15.609508] SELinux: Disabled at runtime. [ 15.738189] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.770925] systemd[1]: Detected virtualization kvm. [ 15.774232] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.467601] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.484383] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.514199] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.537677] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.543829] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.572990] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.599526] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 18.227034] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 18.934415] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.850429] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.033457] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.750562] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.838285] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 25.486931] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 26.131291] NFS: Registering the id_resolver key type [ 26.132920] Key type id_resolver registered [ 26.134046] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg420-server login: [ 39.823356] hrtimer: interrupt took 7416318 ns [ 95.677270] libcfs: loading out-of-tree module taints kernel. [ 95.802927] Key type ._llcrypt registered [ 95.805850] Key type .llcrypt registered [ 95.988894] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 117.094699] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 119.266856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 119.301154] alg: No test for adler32 (adler32-zlib) [ 121.097188] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 122.161443] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 124.015368] Key type lgssc registered [ 126.303587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 143.081092] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 194.813820] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 209.607082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 209.691412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 211.085458] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 211.128240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 211.265908] Lustre: lustre-MDT0000: new disk, initializing [ 211.383352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 211.406941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 216.612904] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.136692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 233.276915] Lustre: 6513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 233.313722] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 233.322634] Lustre: Skipped 1 previous similar message [ 233.452381] Lustre: lustre-MDT0001: new disk, initializing [ 233.609266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 233.670209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 233.686714] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 238.930743] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 246.046459] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 256.673419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 256.934751] Lustre: lustre-OST0000: new disk, initializing [ 256.938625] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 256.947042] Lustre: 8450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 257.019435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 258.362812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 258.372322] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 258.536979] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 264.224302] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 282.378367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 282.473405] Lustre: lustre-OST0001: new disk, initializing [ 282.478484] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 282.482980] Lustre: 9525:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 282.556112] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 291.381783] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 291.883509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 291.896693] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 292.013052] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 305.985450] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 315.997066] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 322.128264] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing check_logdir /tmp/testlogs/ [ 327.824775] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing yml_node [ 332.249590] Lustre: DEBUG MARKER: Client: 2.17.57.112 [ 335.194220] Lustre: DEBUG MARKER: MDS: 2.17.57.112 [ 338.368402] Lustre: DEBUG MARKER: OSS: 2.17.57.112 [ 340.450492] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 00:56:21 EDT 2026 [ 359.594273] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 361.210786] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 377.312819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 377.318579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 377.330469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 379.359501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 379.368342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 379.379483] Lustre: Skipped 1 previous similar message [ 384.481234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 384.488787] Lustre: Skipped 6 previous similar messages [ 389.601333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 389.607925] Lustre: Skipped 3 previous similar messages [ 391.135311] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 391.309417] Lustre: server umount lustre-MDT0000 complete [ 394.697951] LustreError: 9524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843438 with bad export cookie 793076751876023511 [ 394.706107] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 394.713753] LustreError: 9524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 394.725589] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 394.733347] Lustre: Skipped 1 previous similar message [ 394.739701] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 394.746877] Lustre: Skipped 1 previous similar message [ 399.845995] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 399.869751] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 404.961376] LustreError: 6518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 404.962231] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 404.983364] LustreError: 6518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 405.007899] Lustre: Skipped 3 previous similar messages [ 409.055202] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 409.305927] Lustre: server umount lustre-MDT0001 complete [ 413.681578] Lustre: server umount lustre-OST0000 complete [ 417.851593] Lustre: server umount lustre-OST0001 complete [ 425.116976] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 436.214685] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 475.047042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 475.305974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 475.346644] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 475.474461] Lustre: lustre-MDT0000: new disk, initializing [ 475.599402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 475.618284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 480.459299] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 491.844506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 492.051592] Lustre: 15085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 492.114697] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 492.119511] Lustre: Skipped 1 previous similar message [ 492.192459] Lustre: lustre-MDT0001: new disk, initializing [ 492.278199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 492.319031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 492.336798] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 496.971712] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 508.803130] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 516.701811] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 525.773852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 526.221982] Lustre: lustre-OST0000: new disk, initializing [ 526.225936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 526.235379] Lustre: 17053:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 526.357690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 527.676755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 527.693763] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 527.871452] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 533.157181] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 543.586433] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 549.489986] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 549.758272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 556.054813] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 556.227236] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 558.562128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 558.567741] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 558.580465] LustreError: Skipped 1 previous similar message [ 558.584185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 558.598088] Lustre: Skipped 2 previous similar messages [ 572.383368] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 572.598740] Lustre: server umount lustre-OST0000 complete [ 579.044859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.063267] Lustre: Skipped 1 previous similar message [ 582.820442] Lustre: server umount lustre-MDT0000 complete [ 584.161455] LustreError: 15096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.185826] LustreError: 15096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 587.367828] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843631 with bad export cookie 793076751876025044 [ 587.372872] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 587.384946] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 587.916358] Lustre: server umount lustre-MDT0001 complete [ 598.070363] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:00:38 (1788843638) [ 620.651415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 621.213804] LustreError: 19944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.292240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 626.101934] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 626.658318] LustreError: 19945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.785412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.875982] LustreError: 19944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.885608] LustreError: 19944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 642.296616] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 652.025769] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 658.287829] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 660.460660] Lustre: Failing over lustre-MDT0000 [ 660.674716] Lustre: server umount lustre-MDT0000 complete [ 660.966734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 660.969917] LustreError: 20433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 660.989968] Lustre: Skipped 1 previous similar message [ 661.020784] LustreError: 20433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 677.343192] Lustre: 3644:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843705/real 1788843705] req@ffff949382770380 x1875738010754816/t0(0) o400->MGC192.168.204.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788843721 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 677.378577] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 681.280573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 688.610067] LustreError: 3642:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9494869e6a00 x1875738010757504/t0(0) o250->MGC192.168.204.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 688.928689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 688.933229] Lustre: Skipped 1 previous similar message [ 688.995627] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 694.247393] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 694.249078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 694.270105] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 694.367686] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 704.597539] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 709.608981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 709.620240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 709.643035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 709.655979] Lustre: Skipped 7 previous similar messages [ 715.574333] Lustre: server umount lustre-MDT0000 complete [ 716.774407] LustreError: 20870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 716.790926] LustreError: 20870:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 719.792622] LustreError: 20678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843763 with bad export cookie 793076751876026458 [ 719.800398] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 720.173719] Lustre: server umount lustre-MDT0001 complete [ 730.172386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 735.600745] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 745.294979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 751.022111] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 759.347974] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 766.340804] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 768.754953] Lustre: Failing over lustre-MDT0000 [ 769.096972] Lustre: server umount lustre-MDT0000 complete [ 770.855442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 770.870706] Lustre: Skipped 2 previous similar messages [ 781.281910] LustreError: 23283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 781.313173] LustreError: 23283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 787.040981] Lustre: 3643:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843814/real 1788843814] req@ffff9494b2692d80 x1875738010784384/t0(0) o400->MGC192.168.204.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788843830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 787.076048] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 789.085327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 796.349555] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb01931a9b30f404 [ 796.360459] Lustre: MGC192.168.204.120@tcp: Connection restored to 0@lo (at 0@lo) [ 796.368048] Lustre: Skipped 1 previous similar message [ 796.818320] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 801.509477] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 801.770927] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 801.790593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 801.814126] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 813.336805] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 821.222203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 821.232880] Lustre: Skipped 3 previous similar messages [ 824.113183] Lustre: server umount lustre-MDT0000 complete [ 828.346424] LustreError: 23267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843872 with bad export cookie 793076751876027396 [ 828.353647] LustreError: 23267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 828.358609] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 828.756229] Lustre: server umount lustre-MDT0001 complete [ 839.224731] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:04:40 (1788843880) [ 860.637447] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 873.496323] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 921.969592] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 933.735384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 934.141499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 934.215537] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 934.359776] Lustre: lustre-MDT0000: new disk, initializing [ 934.466996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 934.474201] Lustre: Skipped 3 previous similar messages [ 934.491594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 939.866299] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 952.467057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.586318] Lustre: 29663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 952.986357] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 957.918803] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 969.369756] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 976.178664] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 984.480565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 984.851409] Lustre: 31633:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 986.861486] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 986.910671] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 986.976425] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 987.004742] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 992.072759] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1002.827154] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1009.554396] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1033.185811] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1033.200678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1033.209981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1033.230778] Lustre: Skipped 3 previous similar messages [ 1033.249105] Lustre: Skipped 1 previous similar message [ 1038.110474] Lustre: server umount lustre-OST0000 complete [ 1049.540484] Lustre: server umount lustre-MDT0000 complete [ 1053.666410] LustreError: 29676:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1053.712522] LustreError: 29676:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 1054.672160] LustreError: 29655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844098 with bad export cookie 793076751876027809 [ 1054.674405] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1054.679688] LustreError: 29655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1055.158308] Lustre: server umount lustre-MDT0001 complete [ 1070.327371] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1073.683784] Key type lgssc unregistered [ 1074.231729] LNet: 33614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1074.237146] LNetError: 33614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1074.266396] LNet: Removed LNI 192.168.204.120@tcp [ 1075.355188] Key type .llcrypt unregistered [ 1075.356764] Key type ._llcrypt unregistered [ 1101.377930] Key type ._llcrypt registered [ 1101.379956] Key type .llcrypt registered [ 1101.476408] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1102.379466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1102.537962] alg: No test for adler32 (adler32-zlib) [ 1103.582750] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 1103.929510] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1105.658212] Key type lgssc registered [ 1107.173619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1116.669087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1116.685942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1118.088667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1123.628547] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1127.405962] LustreError: 35025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1127.437086] LustreError: 35025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1132.517662] LustreError: 35024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1135.210888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1135.843566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1141.311730] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1151.091846] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1158.427271] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1166.627172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1167.075902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1174.235678] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1180.399552] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1181.894497] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1188.641872] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1202.339225] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:10:43 (1788844243) [ 1205.170529] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1207.705406] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1217.285742] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:10:58 (1788844258) [ 1229.448970] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:11:10 (1788844270) [ 1230.922104] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1240.366644] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1240.372911] Lustre: Skipped 2 previous similar messages [ 1242.561870] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:11:23 (1788844283) [ 1244.798457] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1244.802742] Lustre: Skipped 2 previous similar messages [ 1254.913385] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1254.921053] Lustre: Skipped 4 previous similar messages [ 1272.050770] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:11:53 (1788844313) [ 1275.213936] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1279.078234] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1289.648509] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:12:10 (1788844330) [ 1292.085702] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1292.092885] Lustre: Skipped 6 previous similar messages [ 1308.805636] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:12:29 (1788844349) [ 1327.076854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1327.078761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1327.105412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1327.123023] Lustre: Skipped 1 previous similar message [ 1332.205812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1332.216326] Lustre: Skipped 2 previous similar messages [ 1337.313265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1337.327762] Lustre: Skipped 1 previous similar message [ 1340.383614] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1340.667667] Lustre: server umount lustre-OST0000 complete [ 1346.017486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1346.028720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1346.065274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1347.554379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1351.473608] Lustre: server umount lustre-MDT0000 complete [ 1352.678346] LustreError: 35021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1352.727968] LustreError: 35021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1356.067302] LustreError: 38589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844399 with bad export cookie 10527302385880975785 [ 1356.070196] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1356.093751] LustreError: 38589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1356.681551] Lustre: server umount lustre-MDT0001 complete [ 1370.469755] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1373.755138] Key type lgssc unregistered [ 1374.175799] LNet: 42886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1374.206149] LNetError: 42886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1374.232652] LNet: Removed LNI 192.168.204.120@tcp [ 1375.416518] Key type .llcrypt unregistered [ 1375.420787] Key type ._llcrypt unregistered [ 1397.227793] Key type ._llcrypt registered [ 1397.230906] Key type .llcrypt registered [ 1397.360902] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 1417.705106] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1418.957837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1419.017674] alg: No test for adler32 (adler32-zlib) [ 1420.213116] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 1420.516796] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1422.287495] Key type lgssc registered [ 1423.691606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1469.647990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1469.680177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1470.986899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1471.041699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1471.180593] Lustre: lustre-MDT0000: new disk, initializing [ 1471.291344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1471.338866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1478.272939] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1492.658437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1492.793041] Lustre: 46038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1492.855991] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1492.862077] Lustre: Skipped 1 previous similar message [ 1492.961056] Lustre: lustre-MDT0001: new disk, initializing [ 1493.096354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1493.186047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1493.198816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1498.912276] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1509.846273] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1517.453896] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1526.032133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1526.550298] Lustre: lustre-OST0000: new disk, initializing [ 1526.553570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1526.558358] Lustre: 48009:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1526.686884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1534.064670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1534.082710] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1534.257861] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1534.320051] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1545.705722] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1550.904509] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1551.151586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1557.207462] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1557.565608] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1560.032226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1560.050625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1560.060548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1560.067512] Lustre: Skipped 1 previous similar message [ 1565.153087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1570.283626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1570.291681] Lustre: Skipped 2 previous similar messages [ 1574.367943] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1574.596985] Lustre: server umount lustre-OST0000 complete [ 1580.000829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1580.018783] LustreError: Skipped 1 previous similar message [ 1580.024612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1580.031899] Lustre: Skipped 1 previous similar message [ 1580.035336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1585.633632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1585.645151] Lustre: Skipped 3 previous similar messages [ 1593.319603] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1593.486748] Lustre: server umount lustre-MDT0000 complete [ 1597.906517] LustreError: 46032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844641 with bad export cookie 7980306354880586536 [ 1597.911473] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1597.925511] LustreError: 46032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1598.409401] Lustre: server umount lustre-MDT0001 complete [ 1609.920844] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:17:31 (1788844651) [ 1619.638643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1620.286985] LustreError: 50476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1620.316340] LustreError: 50476:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1620.400378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1625.577476] LustreError: 50477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1626.405508] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1630.690933] LustreError: 50476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1635.820084] LustreError: 50477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1636.695524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1637.114062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1641.858659] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1651.890267] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1659.073836] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1668.093301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1668.534822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1677.972176] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1688.907861] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1698.745704] LustreError: 53019:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1700.101214] Lustre: Modifying parameter general.timeout=1 in log params [ 1701.858363] Lustre: Modifying parameter general.timeout=2 in log params [ 1704.679856] Lustre: Modifying parameter general.timeout=4 in log params [ 1704.691730] Lustre: Skipped 1 previous similar message [ 1709.415969] Lustre: Modifying parameter general.timeout=8 in log params [ 1709.426828] Lustre: Skipped 3 previous similar messages [ 1717.649448] Lustre: Modifying parameter general.timeout=15 in log params [ 1717.659345] Lustre: Skipped 6 previous similar messages [ 1734.730872] Lustre: Modifying parameter general.timeout=30 in log params [ 1734.733363] Lustre: Skipped 14 previous similar messages [ 1767.147710] Lustre: Modifying parameter general.timeout=61 in log params [ 1767.156703] Lustre: Skipped 30 previous similar messages [ 1831.992812] Lustre: Modifying parameter general.timeout=118 in log params [ 1832.001958] Lustre: Skipped 56 previous similar messages [ 1855.231838] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:21:36 (1788844896) [ 1858.937917] Lustre: 59984:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1860.824313] Lustre: 60082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1863.039802] Lustre: 60180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1867.152636] Lustre: 60393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1867.166581] Lustre: 60393:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1901.628638] Lustre: 61861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1901.637113] Lustre: 61861:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1901.676353] LustreError: 61861:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1906.436805] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:22:27 (1788844947) [ 1922.019557] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1922.027920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1922.040554] Lustre: Skipped 2 previous similar messages [ 1922.056031] Lustre: Skipped 4 previous similar messages [ 1927.308354] Lustre: server umount lustre-MDT0000 complete [ 1932.119521] LustreError: 50457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844975 with bad export cookie 7980306354880587418 [ 1932.127459] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1932.146565] LustreError: 50457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1932.258650] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1932.260244] LustreError: 50471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1932.269277] Lustre: Skipped 1 previous similar message [ 1932.793550] Lustre: server umount lustre-MDT0001 complete [ 1938.128972] Lustre: server umount lustre-OST0000 complete [ 1969.854749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1970.165119] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1970.186509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1970.192491] Lustre: Skipped 34 previous similar messages [ 1970.339544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1975.397244] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1985.957946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1986.153308] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1986.213696] Lustre: 64393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1992.375824] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2001.097459] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2007.497309] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2015.401023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2015.766474] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2016.208261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2016.221459] Lustre: Skipped 1 previous similar message [ 2024.319093] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2033.644213] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2035.991829] Lustre: 66960:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 2044.391318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2044.406353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2044.421565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2044.427271] Lustre: Skipped 4 previous similar messages [ 2050.663587] Lustre: server umount lustre-OST0000 complete [ 2058.209589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2058.222490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2058.251961] Lustre: Skipped 1 previous similar message [ 2061.404653] Lustre: server umount lustre-MDT0000 complete [ 2064.864729] LustreError: 66462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2064.891592] LustreError: 66462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2066.491918] LustreError: 64385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845110 with bad export cookie 7980306354880590722 [ 2066.497644] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2066.507418] LustreError: 64385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2067.005953] Lustre: server umount lustre-MDT0001 complete [ 2081.520857] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2085.338523] Key type lgssc unregistered [ 2085.701316] LNet: 68170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2085.713571] LNetError: 68170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2085.753829] LNet: Removed LNI 192.168.204.120@tcp [ 2086.831964] Key type .llcrypt unregistered [ 2086.834133] Key type ._llcrypt unregistered [ 2116.712339] Key type ._llcrypt registered [ 2116.714548] Key type .llcrypt registered [ 2116.804531] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2117.832836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2117.965823] alg: No test for adler32 (adler32-zlib) [ 2118.981761] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 2119.223740] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 2120.904125] Key type lgssc registered [ 2122.018815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2130.815317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2130.830958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2132.458628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2138.262871] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2142.698365] LustreError: 69586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2142.721798] LustreError: 69586:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2147.808902] LustreError: 69585:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2148.484810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2149.288595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2154.312973] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2163.287154] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2170.047896] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2178.508141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2179.085564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2185.924456] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2194.917964] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2213.265876] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2223.078884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2223.084252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2223.093195] Lustre: Skipped 2 previous similar messages [ 2223.104151] Lustre: Skipped 2 previous similar messages [ 2228.192275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2228.201553] Lustre: Skipped 2 previous similar messages [ 2228.854581] Lustre: server umount lustre-MDT0000 complete [ 2232.548297] LustreError: 69566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845276 with bad export cookie 12387142622939942355 [ 2232.550872] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2232.558042] LustreError: 69566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2233.070712] Lustre: server umount lustre-MDT0001 complete [ 2246.576695] Lustre: server umount lustre-OST0000 complete [ 2278.974585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2279.206695] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2279.236350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2279.542621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2284.412685] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2294.475585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2294.624090] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2294.653298] Lustre: 74481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2295.109539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2300.242399] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2310.243692] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2316.304736] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2324.369443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2324.650732] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2325.329561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2333.235292] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2342.173676] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2344.320918] Lustre: 77057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2346.512617] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2355.687086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2355.702043] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2355.732141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2359.564058] Lustre: server umount lustre-OST0000 complete [ 2366.435981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.440701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2366.445978] Lustre: Skipped 2 previous similar messages [ 2366.446739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2366.469349] Lustre: Skipped 3 previous similar messages [ 2371.553162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2371.562759] Lustre: Skipped 1 previous similar message [ 2378.207155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2378.538165] Lustre: server umount lustre-MDT0000 complete [ 2381.795182] LustreError: 76360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.806970] LustreError: 76360:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2382.448635] LustreError: 74474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845426 with bad export cookie 12387142622939943181 [ 2382.451471] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2382.457466] LustreError: 74474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2382.857901] Lustre: server umount lustre-MDT0001 complete [ 2394.626484] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2398.000225] Key type lgssc unregistered [ 2398.336418] LNet: 78266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2398.342847] LNetError: 78266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2398.369614] LNet: Removed LNI 192.168.204.120@tcp [ 2399.561732] Key type .llcrypt unregistered [ 2399.569683] Key type ._llcrypt unregistered [ 2427.728334] Key type ._llcrypt registered [ 2427.730313] Key type .llcrypt registered [ 2427.809498] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2428.713559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2428.737067] alg: No test for adler32 (adler32-zlib) [ 2429.766260] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 2430.041858] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 2431.751193] Key type lgssc registered [ 2432.959283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2440.855175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2440.873602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2442.543876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2447.954844] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2452.977112] LustreError: 79676:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.989385] LustreError: 79676:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2458.084892] LustreError: 79677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.466361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2458.994171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2463.951751] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2471.253320] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2476.285984] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2482.661494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2483.081806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2488.931852] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2496.751256] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2502.244680] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2503.306348] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2505.058280] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2505.063114] Lustre: Skipped 1 previous similar message [ 2507.656466] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2507.662338] Lustre: Skipped 2 previous similar messages [ 2512.013622] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2512.018400] Lustre: Skipped 4 previous similar messages [ 2520.809826] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2520.820502] Lustre: Skipped 9 previous similar messages [ 2536.871206] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2536.878319] Lustre: Skipped 16 previous similar messages [ 2569.001683] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2569.008391] Lustre: Skipped 31 previous similar messages [ 2610.459193] Lustre: 87778:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2613.431979] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 01:34:14 (1788845654) [ 2615.621979] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2618.049730] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 01:34:19 (1788845659) [ 2626.535668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2626.538488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2626.555439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2626.592351] Lustre: Skipped 1 previous similar message [ 2628.716836] Lustre: server umount lustre-OST0000 complete [ 2633.184574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2633.196880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2633.218752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2633.229121] Lustre: Skipped 1 previous similar message [ 2636.789291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2636.791855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2638.562351] Lustre: server umount lustre-MDT0000 complete [ 2641.892239] LustreError: 81585:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.912666] LustreError: 81585:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2642.639874] LustreError: 79656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845686 with bad export cookie 4911236786497302970 [ 2642.647497] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2642.658116] LustreError: 79656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2642.996754] Lustre: server umount lustre-MDT0001 complete [ 2655.705594] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2658.660383] Key type lgssc unregistered [ 2659.024804] LNet: 89026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2659.043341] LNetError: 89026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2659.075244] LNet: Removed LNI 192.168.204.120@tcp [ 2660.024780] Key type .llcrypt unregistered [ 2660.026887] Key type ._llcrypt unregistered [ 2666.325449] Key type ._llcrypt registered [ 2666.333637] Key type .llcrypt registered [ 2666.483652] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2671.789265] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules [ 2672.543859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2672.642056] alg: No test for adler32 (adler32-zlib) [ 2673.669222] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 2673.872706] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 2675.559221] Key type lgssc registered [ 2676.728303] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2677.591776] LustreError: 90224:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2698.787807] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2700.408817] LustreError: 90224:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2709.821729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2709.834944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2711.238322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2716.395912] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2720.741273] LustreError: 91252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2720.760813] LustreError: 91252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2725.865328] LustreError: 91251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2727.676347] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 01:36:08 (1788845768) [ 2730.990921] LustreError: 91252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2733.156426] LustreError: 91283:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2733.172360] LustreError: 91283:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2733.182861] LustreError: 91283:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2733.573365] Lustre: server umount lustre-MDT0000 complete [ 2748.061847] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2750.727257] Key type lgssc unregistered [ 2751.046506] LNet: 92775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2751.062713] LNetError: 92775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2751.085278] LNet: Removed LNI 192.168.204.120@tcp [ 2751.667815] Key type .llcrypt unregistered [ 2751.669945] Key type ._llcrypt unregistered [ 2772.664479] Key type ._llcrypt registered [ 2772.666932] Key type .llcrypt registered [ 2772.795133] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2774.206527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2774.376220] alg: No test for adler32 (adler32-zlib) [ 2775.635496] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 2776.068429] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 2777.831236] Key type lgssc registered [ 2779.765711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2790.236547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2790.284655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2791.992975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2797.736665] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2802.145377] LustreError: 93992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2802.181528] LustreError: 93992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2807.291067] LustreError: 93993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2809.114838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2809.870864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2815.301689] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2823.729085] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2829.807677] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2836.540566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2837.270247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2843.886512] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2853.054419] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2922.109036] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 01:39:23 (1788845963) [ 2924.219364] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2927.037567] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 01:39:27 (1788845967) [ 2932.705432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2932.718694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2932.739863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2933.739912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2933.756085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2933.770924] Lustre: Skipped 1 previous similar message [ 2933.791720] Lustre: Skipped 2 previous similar messages [ 2938.009188] Lustre: server umount lustre-MDT0000 complete [ 2938.849855] LustreError: 94723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2941.780222] LustreError: 93972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845985 with bad export cookie 11589416693348327413 [ 2941.785498] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2941.791506] LustreError: 93972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2942.165108] Lustre: server umount lustre-MDT0001 complete [ 2957.154925] Lustre: server umount lustre-OST0000 complete [ 2969.364868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2969.779382] LustreError: 98313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2969.801589] LustreError: 98313:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2969.964377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2974.920061] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2975.212558] LustreError: 98314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2983.652488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2984.457404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2988.722747] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2997.556428] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3005.664405] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3021.019939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3021.358216] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3021.379171] LustreError: 98300:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3021.427069] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3021.443586] LustreError: 100398:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 3021.447586] LustreError: 100398:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 3021.455590] LustreError: 100398:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3021.541963] Lustre: server umount lustre-OST0000 complete [ 3021.544264] LustreError: 100398:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3031.288699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3031.625771] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3031.634880] LustreError: 98300:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3031.664381] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3031.675134] LustreError: 100864:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 3031.684338] LustreError: 100864:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 3031.690368] LustreError: 100864:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3031.760204] Lustre: server umount lustre-OST0000 complete [ 3031.762922] LustreError: 100864:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3041.629521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3041.863686] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3041.870516] Lustre: Found index 0 for lustre-OST0000, updating log [ 3041.882578] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3041.962521] Lustre: lustre-OST0000: new disk, initializing [ 3041.968396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3041.978504] Lustre: 101400:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3042.231611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3048.960471] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3053.419094] Lustre: Failing over lustre-OST0000 [ 3053.562515] Lustre: server umount lustre-OST0000 complete [ 3059.681834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3059.689455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3059.699175] Lustre: Skipped 1 previous similar message [ 3059.716453] Lustre: Skipped 1 previous similar message [ 3064.393127] Lustre: server umount lustre-MDT0000 complete [ 3064.801661] LustreError: 98314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3064.815644] LustreError: 98314:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3068.475159] LustreError: 100881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846112 with bad export cookie 11589416693348330157 [ 3068.486243] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3068.493268] LustreError: 100881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3068.896507] Lustre: server umount lustre-MDT0001 complete [ 3080.388158] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 01:42:01 (1788846121) [ 3098.675418] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3109.476984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3110.042665] LustreError: 103746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3110.111108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3115.288470] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3124.368792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3130.016840] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3133.083342] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3140.987691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3148.827583] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3156.967957] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3157.066914] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3157.160084] Lustre: lustre-OST0001: new disk, initializing [ 3157.163591] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3157.169526] Lustre: 106065:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3157.553426] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3157.558650] Lustre: Skipped 2 previous similar messages [ 3163.240717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3163.255853] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3163.395036] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3164.587426] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3175.088598] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3183.262289] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3188.959682] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.20@tcp (stopping) [ 3189.220082] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3189.251549] Lustre: Skipped 2 previous similar messages [ 3194.026498] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.20@tcp (stopping) [ 3194.037738] Lustre: Skipped 3 previous similar messages [ 3194.587463] Lustre: server umount lustre-MDT0001 complete [ 3199.139847] LustreError: 103741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3199.157471] LustreError: 103741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3206.035763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3206.291786] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3206.306645] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3206.386556] Lustre: 103735:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3207.551491] LustreError: 107588:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3207.556781] LustreError: 107588:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3207.573690] LustreError: 103738:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3207.725612] LustreError: 107602:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.120@tcp: cfg command failed: rc = -17 [ 3207.746961] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3207.746961] [ 3212.815591] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3212.941461] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3225.701813] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 01:44:26 (1788846266) [ 3247.957940] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3269.853240] Lustre: 109684:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3296.600831] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3301.052990] Lustre: 110820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3396.076785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3396.084827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3396.098482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3396.103453] Lustre: Skipped 3 previous similar messages [ 3399.863314] Lustre: server umount lustre-MDT0000 complete [ 3402.214450] LustreError: 105263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3402.234438] LustreError: 105263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 3404.037701] LustreError: 104886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846447 with bad export cookie 11589416693348331151 [ 3404.046146] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3404.360938] Lustre: server umount lustre-MDT0001 complete [ 3419.739460] Lustre: server umount lustre-OST0000 complete [ 3434.089460] Lustre: server umount lustre-OST0001 complete [ 3439.670743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3450.182388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3460.094247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3470.968032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3493.653090] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3506.102775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3506.116813] Lustre: lustre-MDT0000: reset Object Index mappings [ 3506.856736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3506.858625] Lustre: Skipped 1 previous similar message [ 3511.782913] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3521.301331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3521.325983] Lustre: lustre-MDT0001: reset Object Index mappings [ 3521.506520] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3521.514458] Lustre: Skipped 4 previous similar messages [ 3527.258525] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3530.656721] Lustre: 115925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3537.684782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3545.251763] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3546.682096] LustreError: 116278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3546.711583] LustreError: 116278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3546.726842] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 3552.763485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3555.827307] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3561.980377] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3561.991169] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3565.313164] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3575.716810] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3581.287400] Lustre: 117804:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3642.356900] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 01:51:23 (1788846683) [ 3669.488727] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3669.490453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3669.521241] Lustre: Skipped 5 previous similar messages [ 3669.541334] Lustre: Skipped 3 previous similar messages [ 3672.762947] Lustre: server umount lustre-MDT0000 complete [ 3677.404165] LustreError: 114757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846721 with bad export cookie 11589416693348378023 [ 3677.406826] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3677.415039] LustreError: 114757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3684.080404] Lustre: server umount lustre-MDT0001 complete [ 3704.287287] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3704.469553] Lustre: server umount lustre-OST0000 complete [ 3723.234847] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3723.418932] Lustre: server umount lustre-OST0001 complete [ 3731.044598] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 3744.102762] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3783.918186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3784.179396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3784.225415] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3784.358528] Lustre: lustre-MDT0000: new disk, initializing [ 3784.440371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3784.446906] Lustre: Skipped 3 previous similar messages [ 3784.459161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3789.345941] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3796.539350] Lustre: server umount lustre-MDT0000 complete [ 3806.752351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3807.067115] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3807.089612] Lustre: 122075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3807.131291] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3812.608638] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3818.619088] Lustre: server umount lustre-MDT0000 complete [ 3828.950657] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 01:54:29 (1788846869) [ 3830.970468] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3833.747564] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 01:54:34 (1788846874) [ 3855.623295] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3868.158628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3874.623334] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3884.205192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3884.332030] Lustre: 124128:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3884.413838] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3884.417884] Lustre: Skipped 1 previous similar message [ 3884.511474] Lustre: lustre-MDT0001: new disk, initializing [ 3884.689057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3884.707327] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3889.562246] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3895.149493] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3902.447266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3902.665249] Lustre: lustre-OST0000: new disk, initializing [ 3902.668839] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3902.688444] Lustre: 125709:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3904.315582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3904.334886] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3904.551305] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3911.326510] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3923.927383] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3924.147519] Lustre: lustre-OST0001: new disk, initializing [ 3924.152060] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3924.157764] Lustre: 126580:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3926.034178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3926.046646] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3926.161033] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3931.131639] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3943.963519] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3948.597694] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4405.715503] Lustre: 133020:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4405.725442] Lustre: 133020:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4405.732987] Lustre: 133020:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4405.747706] Lustre: 133020:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4405.769522] Lustre: 133020:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4405.781588] Lustre: 133020:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4734.651920] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:09:35 (1788847775) [ 4749.797325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4749.802462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4749.815058] Lustre: Skipped 3 previous similar messages [ 4749.819128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4749.822628] Lustre: Skipped 4 previous similar messages [ 4752.380559] Lustre: server umount lustre-MDT0000 complete [ 4753.891153] LustreError: 124140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4753.895324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4753.915300] LustreError: 124140:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 4753.935725] Lustre: Skipped 1 previous similar message [ 4756.886253] LustreError: 133080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847800 with bad export cookie 11589416693348448457 [ 4756.895848] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4756.898667] LustreError: 133080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4759.013294] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4759.030412] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4759.037252] Lustre: Skipped 2 previous similar messages [ 4771.295193] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4771.656273] Lustre: server umount lustre-MDT0001 complete [ 4791.263533] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4791.443861] Lustre: server umount lustre-OST0000 complete [ 4801.638591] Lustre: server umount lustre-OST0001 complete [ 4808.564470] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 4819.673588] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 4852.518140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4852.721207] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4852.749594] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4852.840607] Lustre: lustre-MDT0000: new disk, initializing [ 4852.902192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4852.912277] Lustre: Skipped 5 previous similar messages [ 4852.924974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4857.236345] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4868.148261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4868.342907] Lustre: 136081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4868.447617] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4868.459397] Lustre: Skipped 1 previous similar message [ 4868.594492] Lustre: lustre-MDT0001: new disk, initializing [ 4868.759953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4868.771479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4873.535187] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4884.441900] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4890.179657] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4897.758952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4898.226306] Lustre: lustre-OST0000: new disk, initializing [ 4898.240473] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4898.259777] Lustre: 138052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4898.438496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4898.450276] Lustre: Skipped 1 previous similar message [ 4899.760081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4899.771253] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4899.886201] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4906.701765] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4917.129662] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4928.049463] Lustre: lustre-MDD0000: changelog on [ 4931.309036] Lustre: lustre-MDD0001: changelog on [ 6688.101639] Lustre: lustre-MDD0001: changelog off [ 6689.520709] Lustre: 136089:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6690.140995] Lustre: lustre-MDD0000: changelog off [ 6696.930746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6696.936397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6696.937917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6696.950048] Lustre: Skipped 1 previous similar message [ 6696.963341] Lustre: Skipped 8 previous similar messages [ 6700.285409] Lustre: server umount lustre-OST0000 complete [ 6706.670295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6706.677175] LustreError: Skipped 1 previous similar message [ 6706.680770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6706.688246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6709.348672] Lustre: server umount lustre-MDT0000 complete [ 6712.018488] LustreError: 136878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849755 with bad export cookie 11589416693349274513 [ 6712.020059] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6712.035418] LustreError: 136878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6712.267769] Lustre: server umount lustre-MDT0001 complete [ 6720.121490] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 6722.193934] Key type lgssc unregistered [ 6722.502354] LNet: 147161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6722.509897] LNetError: 147161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6722.523821] LNet: Removed LNI 192.168.204.120@tcp [ 6723.211865] Key type .llcrypt unregistered [ 6723.221410] Key type ._llcrypt unregistered [ 6738.206229] Key type ._llcrypt registered [ 6738.209566] Key type .llcrypt registered [ 6738.294356] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 6749.914977] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 6750.511440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6750.523336] alg: No test for adler32 (adler32-zlib) [ 6751.481494] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 6751.649935] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 6753.303731] Key type lgssc registered [ 6754.235749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6779.682487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6779.692284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6780.866922] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6780.893248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6780.961170] Lustre: lustre-MDT0000: new disk, initializing [ 6781.048310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6781.061883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6784.540484] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6792.069858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6792.133606] Lustre: 150336:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6792.180607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6792.186252] Lustre: Skipped 1 previous similar message [ 6792.239777] Lustre: lustre-MDT0001: new disk, initializing [ 6792.308533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6792.336232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6792.344042] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6795.258631] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6802.312856] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6806.621216] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6812.169178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6812.398533] Lustre: lustre-OST0000: new disk, initializing [ 6812.401981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6812.406861] Lustre: 152304:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6812.477931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6817.346303] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6822.476594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6822.495918] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6822.626147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6825.276737] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6830.886524] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_module obdecho/obdecho [ 6832.337519] LustreError: 153306:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6832.350158] LustreError: 153306:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6833.898787] LustreError: 153403:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6843.364991] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6843.370031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6843.388923] Lustre: Skipped 2 previous similar messages [ 6843.397053] Lustre: Skipped 1 previous similar message [ 6843.410465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6848.486227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6848.493118] Lustre: Skipped 3 previous similar messages [ 6848.733888] Lustre: server umount lustre-MDT0000 complete [ 6851.843836] LustreError: 150328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849895 with bad export cookie 17794629803609116015 [ 6851.850749] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6851.860509] LustreError: 150328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6852.104205] Lustre: server umount lustre-MDT0001 complete [ 6865.681690] Lustre: server umount lustre-OST0000 complete [ 6882.842996] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 6890.953131] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 6915.978977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6916.169390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6916.191076] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6916.243482] Lustre: lustre-MDT0000: new disk, initializing [ 6916.297484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6916.311313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6919.308418] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6926.727128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6926.794613] Lustre: 157120:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6926.837587] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6926.843456] Lustre: Skipped 1 previous similar message [ 6926.889098] Lustre: lustre-MDT0001: new disk, initializing [ 6926.953685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6926.978120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6926.991289] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6929.661426] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6936.668300] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6941.177217] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6946.032960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6946.279192] Lustre: lustre-OST0000: new disk, initializing [ 6946.289567] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6946.301233] Lustre: 159085:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6946.396665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6951.466858] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6956.048230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6956.056058] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6956.119435] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6958.431942] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6961.546759] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6961.686541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6964.534425] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6964.688369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6966.246922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6966.251147] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6966.252915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6966.255609] LustreError: Skipped 1 previous similar message [ 6966.280867] Lustre: Skipped 1 previous similar message [ 6971.364081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6971.371534] Lustre: Skipped 2 previous similar messages [ 6972.088490] Lustre: server umount lustre-OST0000 complete [ 6976.482353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6976.484213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6976.490888] Lustre: Skipped 1 previous similar message [ 6976.502107] Lustre: Skipped 1 previous similar message [ 6986.723951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6986.729615] Lustre: Skipped 3 previous similar messages [ 6989.282585] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6989.401368] Lustre: server umount lustre-MDT0000 complete [ 6991.822749] LustreError: 157111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850035 with bad export cookie 17794629803609116974 [ 6991.824795] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6991.833753] LustreError: 157111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6992.067692] Lustre: server umount lustre-MDT0001 complete [ 6994.696995] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 02:47:16 (1788850036) [ 6995.937291] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6997.292762] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 02:47:19 (1788850039) [ 7003.301608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7003.660896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7006.857709] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7008.736576] LustreError: 161496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7008.749560] LustreError: 161496:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7013.101841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7016.958804] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7022.578983] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7027.726672] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7032.980345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7037.801306] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7042.968166] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7058.613332] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.20@tcp (stopping) [ 7059.424526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7059.429613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7060.112543] Lustre: server umount lustre-MDT0000 complete [ 7062.733023] LustreError: 161478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850106 with bad export cookie 17794629803609117856 [ 7062.734232] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7062.740994] LustreError: 161478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7063.713144] LustreError: 161491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7065.056758] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7065.059447] LustreError: 161491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7065.066936] Lustre: Skipped 2 previous similar messages [ 7067.105608] LustreError: 163713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7072.225313] LustreError: 163713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7072.247265] LustreError: 163713:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7077.343198] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7077.478921] Lustre: server umount lustre-MDT0001 complete [ 7081.581173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7090.077653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7090.244982] LustreError: 165361:0:(llog.c:1655:llog_backup()) MGC192.168.204.120@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7090.251134] Lustre: 165361:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.120@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7103.971507] LustreError: 165366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7103.983310] LustreError: 165366:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7120.040170] LustreError: 165368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7120.058086] LustreError: 165368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 7151.071299] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7151.075159] Lustre: 165361:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.120@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7153.632050] LustreError: 165367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7153.648511] LustreError: 165367:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 23 previous similar messages [ 7192.047883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7192.058694] Lustre: Skipped 2 previous similar messages [ 7194.503161] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7197.158671] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7199.538622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7209.839705] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7213.036149] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7213.938500] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7214.874682] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7218.342467] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7219.528162] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7233.510758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7233.511057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7233.526886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7233.530706] Lustre: Skipped 12 previous similar messages [ 7238.014926] Lustre: server umount lustre-MDT0000 complete [ 7238.625950] LustreError: 165384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7238.635542] LustreError: 165384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 39 previous similar messages [ 7240.092332] LustreError: 161477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850283 with bad export cookie 17794629803609124345 [ 7240.093974] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7240.097509] LustreError: 161477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7240.273187] Lustre: server umount lustre-MDT0001 complete [ 7252.672841] Lustre: server umount lustre-OST0000 complete [ 7265.194502] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 7270.722883] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 7290.757223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7290.930581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7290.958090] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7291.011199] Lustre: lustre-MDT0000: new disk, initializing [ 7291.071128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7291.077340] Lustre: Skipped 1 previous similar message [ 7291.088888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7293.777032] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7300.554538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7300.660142] Lustre: 170460:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7300.922378] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7303.459289] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7309.784881] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7313.233612] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7317.884857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7318.093344] Lustre: 172429:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7322.181819] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7324.182815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7324.187920] Lustre: Skipped 1 previous similar message [ 7324.193304] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7324.246887] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7328.208847] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7331.215387] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7331.350191] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7334.253581] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7334.397528] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7339.491104] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7339.491850] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7339.493172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7339.500543] Lustre: Skipped 3 previous similar messages [ 7339.520509] LustreError: Skipped 1 previous similar message [ 7341.627515] Lustre: server umount lustre-OST0000 complete [ 7349.991414] Lustre: server umount lustre-MDT0000 complete [ 7352.236257] LustreError: 170452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850395 with bad export cookie 17794629803609141138 [ 7352.237729] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7352.247835] LustreError: 170452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7352.442927] Lustre: server umount lustre-MDT0001 complete [ 7357.379208] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 02:53:19 (1788850399) [ 7361.385235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7364.224914] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7367.140638] LustreError: 174838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7367.157427] LustreError: 174838:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7369.547506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7372.549446] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7377.335299] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7380.758637] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7384.745055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7388.690410] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7393.131642] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7403.056545] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 02:54:05 (1788850445) [ 7405.537090] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7405.539057] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7405.548709] Lustre: Skipped 2 previous similar messages [ 7411.210539] Lustre: server umount lustre-OST0000 complete [ 7419.641754] Lustre: server umount lustre-MDT0000 complete [ 7421.749312] LustreError: 177812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850465 with bad export cookie 17794629803609142020 [ 7421.755915] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7421.763689] LustreError: 177812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7421.984642] Lustre: server umount lustre-MDT0001 complete [ 7428.005515] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 7429.619779] Key type lgssc unregistered [ 7429.835371] LNet: 178562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7429.843088] LNetError: 178562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7429.854659] LNet: Removed LNI 192.168.204.120@tcp [ 7430.384265] Key type .llcrypt unregistered [ 7430.386221] Key type ._llcrypt unregistered [ 7442.264528] Key type ._llcrypt registered [ 7442.267051] Key type .llcrypt registered [ 7442.349306] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 7442.796302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7442.822618] alg: No test for adler32 (adler32-zlib) [ 7443.782316] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 7443.937933] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 7445.535276] Key type lgssc registered [ 7446.238685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7450.471925] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7450.487794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7466.015395] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7471.135312] LustreError: 179778:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.120@tcp: failed processing log, type 4: rc = -110 [ 7476.255521] LustreError: 179778:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7476.265664] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7476.269414] LustreError: MGC192.168.204.120@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7476.277467] LustreError: 179778:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7476.283369] LustreError: 179778:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7476.287598] LustreError: 179778:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7476.326372] Lustre: server umount lustre-OST0000 complete [ 7476.328778] LustreError: 179778:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7481.273390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7482.521606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7484.976418] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7489.474145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7489.678736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7491.912366] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7495.927955] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7498.923758] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7502.584398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7502.704314] LustreError: 182089:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7502.709428] Lustre: 182089:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.120@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7502.800948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7505.916802] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7509.867406] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7513.057322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7513.058773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7513.063193] Lustre: Skipped 2 previous similar messages [ 7513.066017] Lustre: Skipped 1 previous similar message [ 7515.104782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7515.108924] Lustre: Skipped 2 previous similar messages [ 7520.225466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7525.343116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7525.345094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7525.358517] Lustre: Skipped 4 previous similar messages [ 7525.444298] Lustre: server umount lustre-MDT0000 complete [ 7527.238205] LustreError: 180257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850570 with bad export cookie 6136415850530632687 [ 7527.240883] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7527.247954] LustreError: 180257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7527.345279] LustreError: 181004:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d7f2706e x1875745688281344/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7527.437537] Lustre: server umount lustre-MDT0001 complete [ 7538.482529] Lustre: server umount lustre-OST0000 complete [ 7542.959912] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 7547.938279] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 7567.678813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7567.843496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7567.875877] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7567.932248] Lustre: lustre-MDT0000: new disk, initializing [ 7567.979213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7567.991031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7570.229139] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7576.126192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7576.185163] Lustre: 185612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7576.225919] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7576.230612] Lustre: Skipped 1 previous similar message [ 7576.287273] Lustre: lustre-MDT0001: new disk, initializing [ 7576.347205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7576.368611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7576.375549] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7578.753854] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7584.213422] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7587.031429] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7590.558761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7590.693798] Lustre: lustre-OST0000: new disk, initializing [ 7590.696536] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7590.701158] Lustre: 187579:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7590.749511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7593.628205] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7596.053257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7596.061028] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7596.121944] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7598.964668] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7601.200525] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7601.332202] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7603.780671] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7603.892310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7606.246282] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7606.248702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7606.253935] LustreError: Skipped 1 previous similar message [ 7606.263056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7606.269045] Lustre: Skipped 1 previous similar message [ 7611.062465] Lustre: server umount lustre-OST0000 complete [ 7616.484914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7616.489988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7616.499803] Lustre: Skipped 2 previous similar messages [ 7616.505615] Lustre: Skipped 1 previous similar message [ 7628.255739] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7628.452879] Lustre: server umount lustre-MDT0000 complete [ 7631.845920] LustreError: 185624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7631.874645] LustreError: 185624:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7634.632751] LustreError: 185603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850678 with bad export cookie 6136415850530633394 [ 7634.635767] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7634.665876] LustreError: 185603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7636.166939] Lustre: server umount lustre-MDT0001 complete [ 7653.567292] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 02:58:13 (1788850693) [ 7686.186675] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 7689.913899] Key type lgssc unregistered [ 7690.200497] LNet: 190524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7690.213962] LNetError: 190524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7690.225880] LNet: Removed LNI 192.168.204.120@tcp [ 7691.428409] Key type .llcrypt unregistered [ 7691.430219] Key type ._llcrypt unregistered [ 7710.616679] Key type ._llcrypt registered [ 7710.618734] Key type .llcrypt registered [ 7710.758394] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 7711.570333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7711.682040] alg: No test for adler32 (adler32-zlib) [ 7712.741666] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 7713.067160] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 7714.791201] Key type lgssc registered [ 7715.683215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7723.355485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7723.369685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7724.769449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7729.403064] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7735.276488] LustreError: 191743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7735.295619] LustreError: 191743:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7738.246608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7738.853934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7743.171896] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7751.590117] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7757.263858] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7764.727866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7764.974777] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7764.977201] LustreError: 193607:0:(llog.c:1696:llog_backup()) MGC192.168.204.120@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7764.983379] Lustre: 193607:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.120@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7765.107757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7770.810194] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7778.489446] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7781.859396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7781.864158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7781.881475] Lustre: Skipped 1 previous similar message [ 7781.899098] Lustre: Skipped 2 previous similar messages [ 7783.976068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7783.991394] Lustre: Skipped 2 previous similar messages [ 7787.601914] Lustre: server umount lustre-MDT0000 complete [ 7789.033197] LustreError: 192774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7789.093722] LustreError: 192774:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7794.147316] LustreError: 191743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7794.172417] LustreError: 191743:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7794.282064] LustreError: 191723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850838 with bad export cookie 11564931662703377739 [ 7794.289411] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7794.312470] LustreError: 191723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7796.321482] LustreError: 192473:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006dd8a067 x1875745970353024/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7796.834072] Lustre: server umount lustre-MDT0001 complete [ 7812.424378] Lustre: server umount lustre-OST0000 complete [ 7822.268911] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 7831.888752] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 7865.706760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7866.034280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7866.065443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7866.175525] Lustre: lustre-MDT0000: new disk, initializing [ 7866.302601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7866.328268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7871.097958] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7882.202812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7882.341221] Lustre: 197140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7882.378892] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7882.385337] Lustre: Skipped 1 previous similar message [ 7882.388627] LustreError: 197924:0:(llog.c:1696:llog_backup()) MGC192.168.204.120@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7882.395231] LustreError: 197924:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7882.400722] Lustre: 197924:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.120@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7882.428559] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7882.435943] Lustre: Skipped 1 previous similar message [ 7882.549806] Lustre: lustre-MDT0001: new disk, initializing [ 7882.662547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7882.696587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7882.704891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7887.917817] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7897.706908] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7903.758508] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7910.685876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7910.935346] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7910.941865] Lustre: Skipped 1 previous similar message [ 7910.946202] LustreError: 199088:0:(llog.c:1696:llog_backup()) MGC192.168.204.120@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7910.956418] LustreError: 199088:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7910.961877] Lustre: 199088:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.120@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7910.970521] Lustre: 199088:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7910.998558] Lustre: lustre-OST0000: new disk, initializing [ 7911.011486] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7911.018255] Lustre: 199111:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7911.135514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7918.063167] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7920.170618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7920.182053] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7920.349070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7929.746881] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7935.154753] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7935.393550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7941.180996] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7941.407803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7946.209607] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7946.218281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7946.241401] Lustre: Skipped 2 previous similar messages [ 7949.892650] Lustre: server umount lustre-OST0000 complete [ 7956.450833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7956.471465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7956.484957] Lustre: Skipped 1 previous similar message [ 7956.486368] Lustre: Skipped 1 previous similar message [ 7961.025137] Lustre: server umount lustre-MDT0000 complete [ 7961.569612] LustreError: 197154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7961.593684] LustreError: 197154:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7964.629335] LustreError: 197134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851008 with bad export cookie 11564931662703378453 [ 7964.643987] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7964.647387] LustreError: 197134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7964.949533] Lustre: server umount lustre-MDT0001 complete [ 7973.567954] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 03:03:35 (1788851015) [ 7991.428423] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 8000.891142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8001.297688] LustreError: 202238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8001.336629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8005.697319] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8011.749893] LustreError: 202239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8011.779535] LustreError: 202239:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8015.097424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8021.140951] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8024.374353] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8031.664249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8040.133734] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8052.023205] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8052.316417] Lustre: lustre-OST0001: new disk, initializing [ 8052.319483] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8052.328585] Lustre: 204548:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8052.466307] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8052.474315] Lustre: Skipped 2 previous similar messages [ 8059.792331] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8060.499329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8060.522399] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8060.657932] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8071.517771] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8082.817454] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8085.277883] Lustre: DEBUG MARKER: ADD OST3 [ 8091.387246] loop: module loaded [ 8091.504022] loop0: detected capacity change from 0 to 204800000 [ 8091.783657] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8091.806415] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8091.822480] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8099.780948] loop0: detected capacity change from 0 to 204800000 [ 8099.870932] Lustre: DEBUG MARKER: STOP OST3 [ 8107.019692] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8107.334844] Lustre: lustre-OST0003: new disk, initializing [ 8107.350517] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8107.360502] Lustre: 206604:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8108.857653] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8108.991371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8109.013508] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8109.067908] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8110.731931] Lustre: Failing over lustre-OST0003 [ 8110.905241] LustreError: 206695:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8111.033544] Lustre: server umount lustre-OST0003 complete [ 8114.153356] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8114.170576] LustreError: 203725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8114.171606] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8115.570826] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8126.650488] Lustre: DEBUG MARKER: START OST3 again [ 8134.040858] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8134.486567] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8134.495120] Lustre: Skipped 1 previous similar message [ 8134.528387] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8134.825946] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8136.228099] Lustre: lustre-OST0003: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 8136.236753] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8141.545227] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8163.366603] Lustre: Failing over lustre-OST0003 [ 8163.560602] Lustre: server umount lustre-OST0003 complete [ 8165.355422] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8165.369325] LustreError: 204556:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8165.373941] Lustre: Skipped 2 previous similar messages [ 8165.406995] LustreError: 204556:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 8184.803091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8184.818225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8184.839158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8184.843384] Lustre: Skipped 1 previous similar message [ 8187.720052] Lustre: server umount lustre-MDT0000 complete [ 8191.658067] LustreError: 202220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851235 with bad export cookie 11564931662703379342 [ 8191.681026] LustreError: 202220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8191.688864] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8196.070181] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8196.078691] Lustre: Skipped 4 previous similar messages [ 8198.110790] Lustre: server umount lustre-MDT0001 complete [ 8203.877735] Lustre: server umount lustre-OST0000 complete [ 8209.055595] Lustre: server umount lustre-OST0001 complete [ 8218.215300] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 8229.267587] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 8267.436864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8267.712250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8267.749876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8267.817519] Lustre: lustre-MDT0000: new disk, initializing [ 8267.943841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8267.962402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8272.651570] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8283.933124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8284.113680] Lustre: 212160:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8284.230993] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8284.236148] Lustre: Skipped 1 previous similar message [ 8284.302319] Lustre: lustre-MDT0001: new disk, initializing [ 8284.512288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8284.538455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8289.929172] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8299.938726] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8306.784740] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8314.482865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8314.811239] Lustre: 214130:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8316.182646] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8316.360524] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8321.981886] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8333.055937] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8338.412140] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8338.642353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8343.317955] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8343.562900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8347.106717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8347.117898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8347.129065] Lustre: Skipped 5 previous similar messages [ 8347.136290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8347.164246] Lustre: Skipped 2 previous similar messages [ 8352.328140] Lustre: server umount lustre-OST0000 complete [ 8362.960208] Lustre: server umount lustre-MDT0000 complete [ 8366.562039] LustreError: 213036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8366.583627] LustreError: 213036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 8366.839145] LustreError: 212154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851410 with bad export cookie 11564931662703381638 [ 8366.850246] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8366.851431] LustreError: 212154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8367.234463] Lustre: server umount lustre-MDT0001 complete [ 8371.422291] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 03:10:12 (1788851412) [ 8391.124132] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 8406.585838] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 8447.464886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8447.786138] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8447.816044] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8447.820645] Lustre: Skipped 1 previous similar message [ 8447.903781] Lustre: lustre-MDT0000: new disk, initializing [ 8447.907092] Lustre: Skipped 1 previous similar message [ 8448.041747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8448.049869] Lustre: Skipped 1 previous similar message [ 8453.211420] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8464.087190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8464.235775] Lustre: 218766:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8464.598517] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8469.615488] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8479.411522] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8485.481303] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8493.204306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8493.525225] Lustre: 220736:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8495.447133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8499.952633] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8510.181484] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8515.752132] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8516.125474] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8521.465123] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8521.824413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8526.305022] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8526.320469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8526.321159] LustreError: Skipped 1 previous similar message [ 8526.347018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8526.348487] Lustre: Skipped 4 previous similar messages [ 8526.374207] Lustre: Skipped 5 previous similar messages [ 8530.259245] Lustre: server umount lustre-OST0000 complete [ 8536.552425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8540.763991] Lustre: server umount lustre-MDT0000 complete [ 8541.669409] LustreError: 218772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8541.685144] LustreError: 218772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8544.055405] LustreError: 218759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851587 with bad export cookie 11564931662703382520 [ 8544.065572] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8544.070194] LustreError: 218759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8544.382675] Lustre: server umount lustre-MDT0001 complete [ 8553.186311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8553.664963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8553.672307] Lustre: Skipped 5 previous similar messages [ 8557.721685] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8568.547031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8573.656704] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8582.850118] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8590.858094] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8598.876267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8606.087456] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8614.143765] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8625.121297] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8625.130769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8625.143734] Lustre: Skipped 2 previous similar messages [ 8625.151334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8625.160560] Lustre: Skipped 2 previous similar messages [ 8631.329831] Lustre: server umount lustre-OST0000 complete [ 8642.563298] Lustre: server umount lustre-MDT0000 complete [ 8647.529338] LustreError: 222934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851691 with bad export cookie 11564931662703383402 [ 8647.531218] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8647.567217] LustreError: 222934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8648.008460] Lustre: server umount lustre-MDT0001 complete [ 8753.765141] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8756.462801] Key type lgssc unregistered [ 8756.738352] LNet: 226616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8756.745597] LNetError: 226616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8756.769305] LNet: Removed LNI 192.168.204.120@tcp [ 8757.696403] Key type .llcrypt unregistered [ 8757.702568] Key type ._llcrypt unregistered [ 8783.271434] Key type ._llcrypt registered [ 8783.274574] Key type .llcrypt registered [ 8783.371765] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 8800.184346] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 8802.450687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8802.531659] alg: No test for adler32 (adler32-zlib) [ 8803.645607] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 8803.898412] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 8805.591388] Key type lgssc registered [ 8807.222233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8849.420017] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8849.455799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8850.745730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8850.786350] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8850.894664] Lustre: lustre-MDT0000: new disk, initializing [ 8851.005580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8851.018856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8855.911714] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8867.507756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8867.651928] Lustre: 229966:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8867.727688] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8867.730907] Lustre: Skipped 1 previous similar message [ 8867.818746] Lustre: lustre-MDT0001: new disk, initializing [ 8867.934708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8867.997934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8868.020813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8872.778820] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8884.436734] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8892.822758] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8900.295373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8900.555552] Lustre: lustre-OST0000: new disk, initializing [ 8900.559399] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8900.566371] Lustre: 231934:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8900.653080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8906.763413] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8908.366796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8908.379611] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8908.461436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8917.320450] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8922.206950] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8922.359700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8926.701461] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8926.887691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8929.251306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8929.259623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8929.269070] Lustre: Skipped 1 previous similar message [ 8929.277644] Lustre: Skipped 1 previous similar message [ 8934.369828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8934.387101] Lustre: Skipped 1 previous similar message [ 8934.881571] Lustre: server umount lustre-OST0000 complete [ 8939.491100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8939.493939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8939.513173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8939.524283] Lustre: Skipped 1 previous similar message [ 8944.609046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8944.622409] Lustre: Skipped 2 previous similar messages [ 8945.689985] Lustre: server umount lustre-MDT0000 complete [ 8949.087314] LustreError: 229958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851992 with bad export cookie 7685914868419401072 [ 8949.095117] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8949.374553] Lustre: server umount lustre-MDT0001 complete [ 8973.042663] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 8986.691422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8987.361095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8992.735969] LustreError: 234870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8992.745945] LustreError: 234870:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8993.131883] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8997.865179] LustreError: 234869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9002.978444] LustreError: 234870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9003.463571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9004.153168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9009.792504] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9015.872938] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9025.015419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9025.438110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9032.254784] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9042.809169] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9043.102849] Lustre: lustre-OST0001: new disk, initializing [ 9043.110228] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9043.131581] Lustre: 237179:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9043.311229] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9051.801837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9051.830888] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9052.019987] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9052.839904] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9065.602390] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9070.579363] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9083.776775] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 03:22:05 (1788852125) [ 9085.921427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9085.929744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9085.942664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9087.983208] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9088.002890] Lustre: Skipped 2 previous similar messages [ 9091.977372] Lustre: server umount lustre-MDT0000 complete [ 9093.099630] LustreError: 234869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9093.131363] LustreError: 234869:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9095.674557] LustreError: 234851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852139 with bad export cookie 7685914868419401954 [ 9095.696492] LustreError: 234851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9095.696555] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9095.996122] Lustre: server umount lustre-MDT0001 complete [ 9110.159904] Lustre: server umount lustre-OST0000 complete [ 9124.340812] Lustre: server umount lustre-OST0001 complete [ 9130.889221] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 9141.099823] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 9180.636659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9180.835699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9180.871358] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9180.976898] Lustre: lustre-MDT0000: new disk, initializing [ 9181.150791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9181.186352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9186.136739] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9197.259307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9197.442535] Lustre: 241127:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9197.545247] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9197.553162] Lustre: Skipped 1 previous similar message [ 9197.621819] Lustre: lustre-MDT0001: new disk, initializing [ 9197.739676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9197.751273] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9202.634701] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9213.285864] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9219.389990] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9227.213820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9227.522192] Lustre: lustre-OST0000: new disk, initializing [ 9227.524981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9227.531330] Lustre: 243094:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9229.232767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9229.241921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9229.423246] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9234.210674] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9245.837482] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9252.698593] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9253.002950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9259.957936] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9260.235153] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9265.122764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9265.126258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9265.147679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9265.164023] Lustre: Skipped 5 previous similar messages [ 9268.610244] Lustre: server umount lustre-OST0000 complete [ 9274.849171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9274.869788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9274.926448] Lustre: Skipped 1 previous similar message [ 9279.642429] Lustre: server umount lustre-MDT0000 complete [ 9280.486857] LustreError: 241991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9284.350569] LustreError: 241920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852328 with bad export cookie 7685914868419404096 [ 9284.359061] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9284.362482] LustreError: 241920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9284.772260] Lustre: server umount lustre-MDT0001 complete [ 9296.761860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9297.517742] LustreError: 245313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9297.558356] LustreError: 245313:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9297.662271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9297.682343] Lustre: Skipped 2 previous similar messages [ 9305.207598] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9315.760853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9321.935136] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9330.208755] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9336.011252] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9343.581475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9351.369872] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9358.671348] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9503.719389] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9503.746662] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9503.777669] Lustre: Skipped 1 previous similar message [ 9503.793880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9503.807917] Lustre: Skipped 3 previous similar messages [ 9509.423886] Lustre: server umount lustre-OST0000 complete [ 9516.000848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9516.017256] LustreError: Skipped 1 previous similar message [ 9519.663413] Lustre: server umount lustre-MDT0000 complete [ 9521.120937] LustreError: 246605:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9521.131982] LustreError: 246605:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9523.187704] LustreError: 245294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852566 with bad export cookie 7685914868419404978 [ 9523.208553] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9523.213309] LustreError: 245294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9523.582915] Lustre: server umount lustre-MDT0001 complete [ 9627.929946] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 9631.537622] Key type lgssc unregistered [ 9632.052887] LNet: 248920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9632.067466] LNetError: 248920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9632.098674] LNet: Removed LNI 192.168.204.120@tcp [ 9633.423641] Key type .llcrypt unregistered [ 9633.431502] Key type ._llcrypt unregistered [ 9672.588906] Key type ._llcrypt registered [ 9672.593900] Key type .llcrypt registered [ 9672.777744] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 9693.278692] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 9694.820357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9695.037867] alg: No test for adler32 (adler32-zlib) [ 9696.165565] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 9696.572202] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 9698.343244] Key type lgssc registered [ 9699.732266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9755.510400] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 9768.431601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9768.469909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9769.816302] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9769.865895] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9770.034793] Lustre: lustre-MDT0000: new disk, initializing [ 9770.172240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9770.213631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9775.089224] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9788.808759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9788.968230] Lustre: 253011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9789.094679] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9789.100719] Lustre: Skipped 1 previous similar message [ 9789.238528] Lustre: lustre-MDT0001: new disk, initializing [ 9789.409740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9789.465511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9789.483887] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9795.151903] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9800.621110] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9807.676221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9807.954805] Lustre: lustre-OST0000: new disk, initializing [ 9807.960160] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9807.974935] Lustre: 254646:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9808.077840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9814.733964] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9815.096594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9815.105260] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9815.236933] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9826.295812] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9826.514458] Lustre: lustre-OST0001: new disk, initializing [ 9826.519888] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9826.529949] Lustre: 255518:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9826.647327] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9833.444749] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9836.091543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9836.099409] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9836.209838] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9843.947989] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9852.063886] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9866.208822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9866.214639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9866.246718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9867.239491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9867.245711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9867.252250] Lustre: Skipped 1 previous similar message [ 9867.277909] Lustre: Skipped 1 previous similar message [ 9870.898398] Lustre: server umount lustre-MDT0000 complete [ 9875.582561] LustreError: 256286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852919 with bad export cookie 9336094698874301269 [ 9875.595823] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9875.604573] LustreError: 256286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9876.316736] Lustre: server umount lustre-MDT0001 complete [ 9890.402147] Lustre: server umount lustre-OST0000 complete [ 9893.663109] Lustre: 250497:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788852921/real 1788852921] req@ffff949387905180 x1875748050774912/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788852937 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9893.718184] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9893.743810] Lustre: Skipped 1 previous similar message [ 9895.034869] Lustre: server umount lustre-OST0001 complete [ 9912.843629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9927.248855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9936.983266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9948.164091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9973.527350] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 9986.136760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9986.171205] Lustre: lustre-MDT0000: reset Object Index mappings [ 9986.774165] LustreError: 260529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9986.802776] LustreError: 260529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9986.879592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9991.802984] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9992.163854] LustreError: 260530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9997.283173] LustreError: 260529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10000.361189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10000.383604] Lustre: lustre-MDT0001: reset Object Index mappings [10001.000398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10005.538596] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10008.715934] Lustre: 261676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10016.325778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10025.239537] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10026.098753] LustreError: 262030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10031.589707] LustreError: 262424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10032.627700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10036.518517] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10037.116051] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10037.128274] Lustre: Skipped 1 previous similar message [10042.394691] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10045.220644] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10053.459042] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10057.030485] Lustre: 263547:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10062.306653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10062.334420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10062.368479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10062.381737] Lustre: Skipped 2 previous similar messages [10064.300678] Lustre: server umount lustre-MDT0000 complete [10067.940483] LustreError: 260525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10067.975121] LustreError: 260525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10069.128969] LustreError: 261678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853112 with bad export cookie 9336094698874308059 [10069.135160] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10069.142932] LustreError: 261678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10069.515465] Lustre: server umount lustre-MDT0001 complete [10083.564438] Lustre: server umount lustre-OST0000 complete [10098.457434] Lustre: server umount lustre-OST0001 complete [10123.593588] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:39:24 (1788853164) [10146.770842] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [10162.507944] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [10210.835940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10211.015876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10211.034654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10211.086382] Lustre: lustre-MDT0000: new disk, initializing [10211.156639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10211.171907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10216.989576] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10229.876945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10230.066317] Lustre: 267831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10230.145428] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10230.159731] Lustre: Skipped 1 previous similar message [10230.293368] Lustre: lustre-MDT0001: new disk, initializing [10230.463427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10230.473892] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10235.707911] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10246.346685] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10252.552633] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10260.744263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10261.092494] Lustre: lustre-OST0000: new disk, initializing [10261.097614] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10261.109452] Lustre: 269799:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10263.196614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10263.213102] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10263.392627] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10268.023458] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10276.649826] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10281.492165] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10281.647881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10287.330948] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10287.571437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10292.198996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10292.216285] Lustre: Skipped 3 previous similar messages [10292.223752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10292.232162] Lustre: Skipped 4 previous similar messages [10296.400432] Lustre: server umount lustre-OST0000 complete [10301.921625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10301.939643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10301.956674] Lustre: Skipped 1 previous similar message [10301.967457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10307.553496] LustreError: 267839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10307.607535] Lustre: server umount lustre-MDT0000 complete [10312.132108] LustreError: 267823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853355 with bad export cookie 9336094698874310145 [10312.148114] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10312.150278] LustreError: 267823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10312.726148] Lustre: server umount lustre-MDT0001 complete [10345.640124] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [10356.511209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10356.931130] LustreError: 272756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10356.953728] LustreError: 272756:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10356.998409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10357.006571] Lustre: Skipped 2 previous similar messages [10362.558730] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10371.203741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10371.554133] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10371.569477] Lustre: Skipped 2 previous similar messages [10377.216619] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10381.016035] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10389.790195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10397.603389] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10408.261637] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10408.573746] Lustre: lustre-OST0001: new disk, initializing [10408.586677] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10408.597321] Lustre: 275062:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10410.277596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10410.307995] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10410.502756] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10417.738759] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10431.132189] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10436.913567] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10443.615613] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10443.740205] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10453.985925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10453.995536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10454.008590] Lustre: Skipped 1 previous similar message [10454.020440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10457.817266] Lustre: server umount lustre-MDT0000 complete [10461.672576] LustreError: 275354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10461.721165] LustreError: 275354:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10462.655276] LustreError: 275829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853506 with bad export cookie 9336094698874311034 [10462.663444] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10462.665226] LustreError: 275829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10463.330801] Lustre: server umount lustre-MDT0001 complete [10478.976598] Lustre: server umount lustre-OST0000 complete [10483.167137] Lustre: 250497:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788853510/real 1788853510] req@ffff94938d88b480 x1875748050942080/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788853526 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10483.646793] Lustre: server umount lustre-OST0001 complete [10505.519312] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [10516.429493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10517.188216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10517.210832] Lustre: Skipped 3 previous similar messages [10522.340567] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10534.326041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10541.398296] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10544.972411] Lustre: 279264:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10554.922587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10562.015755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10565.677326] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10577.996644] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10583.606300] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10587.597594] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10596.896644] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10601.037083] Lustre: 281143:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10607.103297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10607.111230] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10618.574854] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:47:39 (1788853659) [10637.291699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10637.306152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10637.331726] Lustre: Skipped 4 previous similar messages [10637.341234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10637.350197] Lustre: Skipped 4 previous similar messages [10643.311252] Lustre: server umount lustre-MDT0000 complete [10644.966145] LustreError: 278123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10645.003679] LustreError: 278123:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [10649.078487] LustreError: 279642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853692 with bad export cookie 9336094698874317565 [10649.081619] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10649.087730] LustreError: 279642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10655.909127] Lustre: server umount lustre-MDT0001 complete [10667.728694] Lustre: server umount lustre-OST0000 complete [10676.391756] Lustre: server umount lustre-OST0001 complete [10689.015920] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [10704.527518] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [10767.070901] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [10779.532144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10779.766540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10779.796620] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10779.896412] Lustre: lustre-MDT0000: new disk, initializing [10779.982845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10779.988835] Lustre: Skipped 3 previous similar messages [10780.001545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10784.712821] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10796.647291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10796.775876] Lustre: 285413:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10797.214935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10802.560697] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10817.648858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10817.998437] Lustre: 287046:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10819.780623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10825.730917] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10837.662800] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10837.967582] Lustre: 287917:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10839.305658] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10839.312180] Lustre: Skipped 1 previous similar message [10839.557671] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10845.265332] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10857.638100] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10861.425363] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10861.443641] Lustre: Skipped 1 previous similar message [10878.287600] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 03:51:59 (1788853919) [10882.892434] Lustre: server umount lustre-MDT0000 complete [10883.581896] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10883.625794] Lustre: Skipped 6 previous similar messages [10887.104783] LustreError: 285406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853930 with bad export cookie 9336094698874325195 [10887.106444] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10887.116093] LustreError: 285406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10887.418220] Lustre: server umount lustre-MDT0001 complete [10901.700473] Lustre: server umount lustre-OST0000 complete [10904.501792] Lustre: 250497:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788853932/real 1788853932] req@ffff949383b50000 x1875748051108096/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788853948 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10906.602350] Lustre: server umount lustre-OST0001 complete [10915.458907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10915.922381] LustreError: 290164:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10915.940010] LustreError: 290164:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [10920.929493] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10932.272851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10940.782861] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10954.369509] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10961.360688] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10970.760141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10980.639804] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10993.360868] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11016.940235] loop1: detected capacity change from 0 to 204800000 [11017.271193] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11017.288222] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11017.300291] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11022.632816] loop1: detected capacity change from 0 to 204800000 [11029.181204] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11029.375731] LustreError: 290153:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs12042: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11029.388620] LustreError: 290153:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs12042 [11029.393581] LustreError: 290153:0:(mgs_handler.c:517:mgs_target_reg()) fs12042-MDT0001: Error checking target index: rc = -13 [11029.403904] LustreError: fs12042-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11029.417302] LustreError: 293890:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11029.424786] LustreError: 293890:0:(tgt_mount.c:2132:server_put_super()) no obd fs12042-MDT0001 [11029.429679] LustreError: 293890:0:(tgt_mount.c:116:server_deregister_mount()) fs12042-MDT0001 not registered [11029.478063] Lustre: server umount fs12042-MDT0001 complete [11029.483093] LustreError: 293890:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11039.083476] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11039.362407] Lustre: Setting parameter fs12042-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs12042-MDT0001 [11039.396801] Lustre: srv-fs12042-MDT0001: No data found on store. Initialize space. [11039.399815] Lustre: Skipped 4 previous similar messages [11039.469363] Lustre: fs12042-MDT0001: new disk, initializing [11039.471634] Lustre: Skipped 3 previous similar messages [11039.566919] Lustre: srv-fs12042-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11039.582928] Lustre: 294356:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12042-MDT0001-mdtlov: can't create new llog: rc = -115 [11039.592817] LustreError: 294356:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12042-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11045.045321] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11055.106397] Lustre: server umount fs12042-MDT0001 complete [11057.711346] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11077.580426] loop1: detected capacity change from 0 to 204800000 [11085.135308] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11085.787564] Lustre: 295926:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12042-MDT0001-mdtlov: can't create new llog: rc = -115 [11085.796652] LustreError: 295926:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12042-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11092.102273] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11099.154171] Lustre: server umount fs12042-MDT0001 complete [11113.954843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11113.970456] Lustre: Skipped 5 previous similar messages [11113.974640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11113.979133] Lustre: Skipped 10 previous similar messages [11119.281934] Lustre: server umount lustre-MDT0000 complete [11131.110393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11131.357887] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11137.015664] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11137.152394] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11145.791134] loop1: detected capacity change from 0 to 204800000 [11152.945673] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11153.496079] Lustre: 298373:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12042-MDT0001-mdtlov: can't create new llog: rc = -115 [11153.503716] LustreError: 298373:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12042-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11158.460702] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11163.929855] Lustre: server umount fs12042-MDT0001 complete [11192.800928] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11192.806830] LustreError: Skipped 1 previous similar message [11197.319654] Lustre: server umount lustre-OST0000 complete [11216.351200] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11216.731823] Lustre: server umount lustre-MDT0000 complete [11221.577853] LustreError: 290146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854265 with bad export cookie 9336094698874327876 [11221.586588] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11221.603607] LustreError: 290146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11222.066583] Lustre: server umount lustre-MDT0001 complete [11236.783282] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [11240.854938] Key type lgssc unregistered [11241.221510] LNet: 300717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11241.229457] LNetError: 300717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11241.254856] LNet: Removed LNI 192.168.204.120@tcp [11242.613348] Key type .llcrypt unregistered [11242.619325] Key type ._llcrypt unregistered [11269.537334] Key type ._llcrypt registered [11269.539790] Key type .llcrypt registered [11269.646753] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [11270.915151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11271.115705] alg: No test for adler32 (adler32-zlib) [11272.486398] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [11272.900740] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [11274.561622] Key type lgssc registered [11276.293719] Lustre: Echo OBD driver; http://www.lustre.org/ [11287.317716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11287.343331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11289.268959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11295.672320] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11298.822502] LustreError: 301989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11298.858597] LustreError: 301989:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11303.912721] LustreError: 301990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11306.134383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11306.732374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11313.674506] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11323.638879] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11329.989888] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11337.825483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11338.323753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11345.405160] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11351.713130] LustreError: 303820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11354.060124] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11357.157955] LustreError: 303820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11364.332949] LustreError: 304443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11364.386128] LustreError: 304443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11374.568967] LustreError: 304443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11374.587311] LustreError: 304443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11378.984215] loop1: detected capacity change from 0 to 204800000 [11379.116530] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11379.139760] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11379.160059] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11385.348526] loop1: detected capacity change from 0 to 204800000 [11391.278476] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11391.445134] Lustre: fs22758-OST0000: new disk, initializing [11391.449088] Lustre: srv-fs22758-OST0000: No data found on store. Initialize space. [11391.456788] Lustre: 305726:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11391.532604] Lustre: fs22758-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11391.657666] LustreError: 304522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11391.703756] LustreError: 304522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11398.353612] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11407.389232] Lustre: server umount fs22758-OST0000 complete [11424.467724] loop1: detected capacity change from 0 to 204800000 [11425.786332] LustreError: 303820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11425.833687] LustreError: 303820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [11430.890960] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11432.423858] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11432.895611] Lustre: fs22758-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11436.015073] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11442.466259] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11453.543310] loop2: detected capacity change from 0 to 204800000 [11453.802081] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11453.816265] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11453.825415] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11460.859535] loop2: detected capacity change from 0 to 204800000 [11467.826231] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11467.922510] LustreError: fs22758-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11467.929953] LustreError: fs22758-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11467.944387] LustreError: 308780:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11467.955616] LustreError: 308780:0:(tgt_mount.c:2132:server_put_super()) no obd fs22758-OST0001 [11467.961367] LustreError: 308780:0:(tgt_mount.c:116:server_deregister_mount()) fs22758-OST0001 not registered [11467.984933] Lustre: server umount fs22758-OST0001 complete [11467.987770] LustreError: 308780:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11478.611373] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11478.928717] Lustre: fs22758-OST0001: new disk, initializing [11478.935929] Lustre: srv-fs22758-OST0001: No data found on store. Initialize space. [11478.941994] Lustre: 309332:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11479.051434] Lustre: fs22758-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11485.805060] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11491.312037] LustreError: 303821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11491.337473] LustreError: 303821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11494.670850] Lustre: server umount fs22758-OST0000 complete [11503.986970] Lustre: server umount fs22758-OST0001 complete [11517.208894] loop1: detected capacity change from 0 to 204800000 [11523.096611] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11523.321372] Lustre: fs22758-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11530.771686] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11539.560972] loop2: detected capacity change from 0 to 204800000 [11546.968750] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11556.078435] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11565.824885] Lustre: server umount fs22758-OST0000 complete [11576.384692] Lustre: server umount fs22758-OST0001 complete [11583.482977] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11588.597369] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11592.677142] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11592.691664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11592.714060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11593.700821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11593.705422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11597.214165] Lustre: server umount lustre-OST0000 complete [11603.316622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11603.343748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11603.367727] Lustre: Skipped 2 previous similar messages [11608.545119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11608.552985] Lustre: Skipped 1 previous similar message [11609.809114] Lustre: server umount lustre-MDT0000 complete [11614.628780] LustreError: 301970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854658 with bad export cookie 6490200545984919937 [11614.637758] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11614.644538] LustreError: 301970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11615.271492] Lustre: server umount lustre-MDT0001 complete [11628.839538] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [11632.087476] Key type lgssc unregistered [11632.473164] LNet: 315257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11632.486943] LNetError: 315257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11632.515015] LNet: Removed LNI 192.168.204.120@tcp [11633.409079] Key type .llcrypt unregistered [11633.410775] Key type ._llcrypt unregistered [11669.752416] Key type ._llcrypt registered [11669.757183] Key type .llcrypt registered [11669.951166] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [11687.614867] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [11689.307269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11689.344048] alg: No test for adler32 (adler32-zlib) [11690.546206] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [11690.771842] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [11692.467976] Key type lgssc registered [11693.907880] Lustre: Echo OBD driver; http://www.lustre.org/ [11749.593589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11749.615565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11750.879403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11750.903825] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11750.949420] Lustre: lustre-MDT0000: new disk, initializing [11750.997883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11751.013663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11756.267826] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11769.449838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11769.631777] Lustre: 319120:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11769.740180] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11769.759417] Lustre: Skipped 1 previous similar message [11769.855835] Lustre: lustre-MDT0001: new disk, initializing [11769.994365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11770.069932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11770.093346] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11775.462385] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11787.133809] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11794.341304] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11803.016193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11803.497787] Lustre: lustre-OST0000: new disk, initializing [11803.505877] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11803.519405] Lustre: 321087:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11803.678581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11809.391902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11809.415196] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11809.548674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11810.569202] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11820.812388] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11823.057611] Lustre: Mounted lustre-client - version 2.17.57_112_g92cfe64 [11824.244337] Lustre: Unmounted lustre-client [11835.360390] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11835.362463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11835.386267] LustreError: Skipped 1 previous similar message [11835.400951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11835.425706] Lustre: Skipped 1 previous similar message [11838.434275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11838.447817] Lustre: Skipped 1 previous similar message [11841.359033] Lustre: server umount lustre-OST0000 complete [11846.627544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11846.642868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11846.658172] Lustre: Skipped 1 previous similar message [11846.666591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11848.672455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11848.673953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11848.705461] Lustre: Skipped 1 previous similar message [11851.688256] Lustre: server umount lustre-MDT0000 complete [11855.945637] LustreError: 319113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854899 with bad export cookie 14549994339477474766 [11855.981503] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11855.996307] LustreError: 319113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11856.360393] Lustre: server umount lustre-MDT0001 complete [11869.444667] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [11872.036849] LNet: 1 local NIs in recovery (showing 1): 192.168.204.120@tcp [11872.447422] Key type lgssc unregistered [11872.821176] LNet: 323253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11872.835745] LNetError: 323253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11953.765813] LNet: Removed LNI 192.168.204.120@tcp [11954.722416] Key type .llcrypt unregistered [11954.724604] Key type ._llcrypt unregistered [11974.465336] Key type ._llcrypt registered [11974.468485] Key type .llcrypt registered [11974.662562] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [11991.528669] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [11992.635240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11992.659326] alg: No test for adler32 (adler32-zlib) [11993.808523] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [11994.086332] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [11995.775189] Key type lgssc registered [11996.980266] Lustre: Echo OBD driver; http://www.lustre.org/ [12028.388658] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 04:11:09 (1788855069) [12035.649943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12035.690207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12036.971275] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12037.055374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12037.189065] Lustre: lustre-MDT0000: new disk, initializing [12037.331756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12037.360465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12042.390744] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12054.955936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12055.083963] Lustre: 326472:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12055.135166] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12055.139031] Lustre: Skipped 1 previous similar message [12055.198329] Lustre: lustre-MDT0001: new disk, initializing [12055.298293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12055.341791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12055.350988] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12061.318964] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12074.666754] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12083.595029] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12092.794450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12093.258238] Lustre: lustre-OST0000: new disk, initializing [12093.263076] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12093.269608] Lustre: 328442:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12093.414743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12099.721874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12099.727744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12101.722741] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12105.230238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12112.133213] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12130.787915] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12130.790383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12130.796283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12130.799587] Lustre: Skipped 1 previous similar message [12134.343886] Lustre: server umount lustre-OST0000 complete [12141.028702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12141.041987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12141.047180] Lustre: Skipped 2 previous similar messages [12145.691381] Lustre: server umount lustre-MDT0000 complete [12150.344788] LustreError: 326466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855194 with bad export cookie 377560009063713410 [12150.348243] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12150.355401] LustreError: 326466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12150.902893] Lustre: server umount lustre-MDT0001 complete [12165.269845] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [12169.092687] Key type lgssc unregistered [12169.540793] LNet: 330351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12169.557101] LNetError: 330351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12169.595483] LNet: Removed LNI 192.168.204.120@tcp [12170.766902] Key type .llcrypt unregistered [12170.770254] Key type ._llcrypt unregistered [12196.918371] Key type ._llcrypt registered [12196.920392] Key type .llcrypt registered [12197.032552] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [12198.721046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12198.802095] alg: No test for adler32 (adler32-zlib) [12200.202725] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [12200.665433] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [12202.415564] Key type lgssc registered [12204.233089] Lustre: Echo OBD driver; http://www.lustre.org/ [12214.536939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12214.574917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12216.151514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12221.361525] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12225.506668] LustreError: 331622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12225.545802] LustreError: 331622:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12230.625713] LustreError: 331621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12230.679658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12231.060155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12235.966992] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12243.207352] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12248.926979] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12255.884940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12256.205251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12261.785442] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12269.651528] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12285.546805] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12287.580800] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12292.599132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12292.610180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12292.636970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12293.333236] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.20@tcp (stopping) [12296.166437] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12296.167250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12296.186919] Lustre: Skipped 1 previous similar message [12296.201024] Lustre: Skipped 1 previous similar message [12298.415937] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.20@tcp (stopping) [12298.427424] Lustre: Skipped 1 previous similar message [12303.543885] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.20@tcp (stopping) [12303.548495] Lustre: Skipped 3 previous similar messages [12306.400280] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12306.723182] Lustre: server umount lustre-MDT0000 complete [12308.650130] LustreError: 333708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12311.312617] LustreError: 331603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855355 with bad export cookie 6414584958727319606 [12311.340814] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12311.526031] LustreError: 331615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12311.544818] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12311.581488] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12311.585871] Lustre: Skipped 3 previous similar messages [12316.642505] LustreError: 331617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12316.660816] LustreError: 331617:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12317.849100] Lustre: server umount lustre-MDT0001 complete [12322.095520] Lustre: server umount lustre-OST0000 complete [12330.673936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12337.387834] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12340.730359] Lustre: 335807:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.120@tcp [12343.724329] Lustre: server umount MGS complete [12355.520551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12355.937869] LustreError: 336379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12356.003192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12361.793205] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12372.671156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12373.160907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12378.712821] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12389.285596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12389.590895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12397.363941] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12407.011571] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12409.775207] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12416.825814] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12419.029147] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12425.191601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12425.201528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12425.227856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12425.239204] Lustre: Skipped 2 previous similar messages [12430.803524] Lustre: server umount lustre-OST0000 complete [12436.961380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12436.990395] Lustre: Skipped 2 previous similar messages [12441.764434] Lustre: server umount lustre-MDT0000 complete [12442.080212] LustreError: 336379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12442.104911] LustreError: 336379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12445.820281] LustreError: 336360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855489 with bad export cookie 6414584958727320446 [12445.837258] LustreError: 336360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12445.839393] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12446.506874] Lustre: server umount lustre-MDT0001 complete [12459.605764] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [12462.975831] Key type lgssc unregistered [12463.270235] LNet: 339816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12463.283938] LNetError: 339816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12464.295274] LNet: Removed LNI 192.168.204.120@tcp [12465.342259] Key type .llcrypt unregistered [12465.345123] Key type ._llcrypt unregistered [12486.418137] Key type ._llcrypt registered [12486.422050] Key type .llcrypt registered [12486.645763] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [12504.839371] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [12506.413784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12506.489291] alg: No test for adler32 (adler32-zlib) [12507.658751] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [12507.996987] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [12509.751212] Key type lgssc registered [12511.185696] Lustre: Echo OBD driver; http://www.lustre.org/ [12551.830308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12551.859831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12553.230189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12553.287574] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12553.424436] Lustre: lustre-MDT0000: new disk, initializing [12553.567911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12553.606316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12559.445865] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12573.996050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12574.207332] Lustre: 342974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12574.274626] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12574.287939] Lustre: Skipped 1 previous similar message [12574.446851] Lustre: lustre-MDT0001: new disk, initializing [12574.647869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12574.751532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12574.766929] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12581.502722] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12591.273726] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12597.074651] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12604.867292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12605.225751] Lustre: lustre-OST0000: new disk, initializing [12605.229335] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12605.238121] Lustre: 344943:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12605.318067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12611.440449] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12614.828353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12614.853485] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12615.022377] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12623.336315] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12629.209759] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12629.571474] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12635.359621] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12635.902442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12640.745202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12640.761281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12640.810485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12640.819379] Lustre: Skipped 1 previous similar message [12644.740593] Lustre: server umount lustre-OST0000 complete [12650.982425] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12650.990659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12651.012751] Lustre: Skipped 2 previous similar messages [12651.042358] Lustre: Skipped 1 previous similar message [12656.099913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12656.116952] Lustre: Skipped 1 previous similar message [12656.821698] Lustre: server umount lustre-MDT0000 complete [12661.590400] LustreError: 344942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855705 with bad export cookie 4459396387850488925 [12661.597322] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12661.615767] LustreError: 344942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12662.397464] Lustre: server umount lustre-MDT0001 complete [12673.803447] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 04:21:54 (1788855714) [12697.159671] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [12711.603365] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [12752.610373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12752.831122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12752.852217] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12752.910703] Lustre: lustre-MDT0000: new disk, initializing [12752.976150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12752.988694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12759.071291] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12774.032655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12774.200772] Lustre: 349726:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12774.252876] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12774.257303] Lustre: Skipped 1 previous similar message [12774.329679] Lustre: lustre-MDT0001: new disk, initializing [12774.412814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12774.455914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12774.465761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12779.622228] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12790.213677] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12796.440845] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12805.232133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12805.587539] Lustre: lustre-OST0000: new disk, initializing [12805.600766] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12805.618060] Lustre: 351698:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12805.767951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12813.507446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12813.542124] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12813.733792] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12814.902440] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12827.665321] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12875.232208] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12875.234889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12875.251314] LustreError: Skipped 2 previous similar messages [12875.260663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12875.271568] Lustre: Skipped 1 previous similar message [12875.305980] Lustre: Skipped 1 previous similar message [12879.507493] Lustre: server umount lustre-OST0000 complete [12885.478637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12885.512696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12885.526550] Lustre: Skipped 1 previous similar message [12891.245829] Lustre: server umount lustre-MDT0000 complete [12895.399474] LustreError: 351697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855939 with bad export cookie 4459396387850489807 [12895.404777] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12895.420124] LustreError: 351697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12895.878811] Lustre: server umount lustre-MDT0001 complete [12907.610899] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [12910.624073] Key type lgssc unregistered [12910.915547] LNet: 353871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12910.925854] LNetError: 353871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12910.946577] LNet: Removed LNI 192.168.204.120@tcp [12911.875404] Key type .llcrypt unregistered [12911.878421] Key type ._llcrypt unregistered [13026.973309] Key type ._llcrypt registered [13026.976203] Key type .llcrypt registered [13027.093072] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13027.411343] Key type .llcrypt unregistered [13027.412693] Key type ._llcrypt unregistered [13115.617746] Key type ._llcrypt registered [13115.621063] Key type .llcrypt registered [13115.862389] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13116.541354] Key type .llcrypt unregistered [13116.552382] Key type ._llcrypt unregistered [13155.046596] Key type ._llcrypt registered [13155.048687] Key type .llcrypt registered [13155.191812] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13155.489784] Key type .llcrypt unregistered [13155.502250] Key type ._llcrypt unregistered [13186.455517] Key type ._llcrypt registered [13186.458808] Key type .llcrypt registered [13186.592643] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13186.868448] Key type .llcrypt unregistered [13186.871112] Key type ._llcrypt unregistered [13278.373565] Key type ._llcrypt registered [13278.376826] Key type .llcrypt registered [13278.501522] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13278.893736] Key type .llcrypt unregistered [13278.895720] Key type ._llcrypt unregistered [13318.289917] Key type ._llcrypt registered [13318.292972] Key type .llcrypt registered [13318.420544] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13318.818308] Key type .llcrypt unregistered [13318.820243] Key type ._llcrypt unregistered [13352.955339] Key type ._llcrypt registered [13352.960303] Key type .llcrypt registered [13353.133830] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13353.431801] Key type .llcrypt unregistered [13353.433828] Key type ._llcrypt unregistered [13389.792510] Key type ._llcrypt registered [13389.797361] Key type .llcrypt registered [13389.917475] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13390.176311] Key type .llcrypt unregistered [13390.180819] Key type ._llcrypt unregistered [13433.773534] Key type ._llcrypt registered [13433.776734] Key type .llcrypt registered [13433.897130] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13434.127531] Key type .llcrypt unregistered [13434.137445] Key type ._llcrypt unregistered [13469.174819] Key type ._llcrypt registered [13469.177151] Key type .llcrypt registered [13469.269430] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13469.580382] Key type .llcrypt unregistered [13469.583235] Key type ._llcrypt unregistered [13537.443574] Key type ._llcrypt registered [13537.455292] Key type .llcrypt registered [13537.573621] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13537.868343] Key type .llcrypt unregistered [13537.871910] Key type ._llcrypt unregistered [13574.748778] Key type ._llcrypt registered [13574.753207] Key type .llcrypt registered [13575.006463] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13575.517413] Key type .llcrypt unregistered [13575.519113] Key type ._llcrypt unregistered [13616.527327] Key type ._llcrypt registered [13616.529787] Key type .llcrypt registered [13616.712902] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13617.185452] Key type .llcrypt unregistered [13617.187683] Key type ._llcrypt unregistered [13673.015281] Key type ._llcrypt registered [13673.016987] Key type .llcrypt registered [13673.154274] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [13690.763959] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [13692.122021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13692.379335] alg: No test for adler32 (adler32-zlib) [13693.662777] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [13694.021921] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [13695.791548] Key type lgssc registered [13697.519956] Lustre: Echo OBD driver; http://www.lustre.org/ [13743.404879] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13743.466812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13744.756959] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13744.810861] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13744.987636] Lustre: tst250fs-MDT0000: new disk, initializing [13745.080922] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13745.117714] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13750.953407] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13764.169833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13764.454667] Lustre: 374167:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13764.609664] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13764.613964] Lustre: Skipped 1 previous similar message [13764.757943] Lustre: tst250fs-MDT0001: new disk, initializing [13764.949750] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13765.048916] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13765.068704] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13770.541474] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13776.485625] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13784.392995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13784.662295] Lustre: tst250fs-OST0000: new disk, initializing [13784.665418] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13784.680328] Lustre: 375802:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13784.825083] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13785.840304] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13785.855666] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13786.053207] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13791.790352] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13804.235887] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13804.555239] Lustre: tst250fs-OST0001: new disk, initializing [13804.565090] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13804.576550] Lustre: 376674:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13804.725557] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13810.381493] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13810.394838] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13810.568850] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13813.132444] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13820.476831] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13823.139439] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13823.168899] Lustre: Skipped 1 previous similar message [13846.505706] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13846.514441] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13846.545214] Lustre: Skipped 3 previous similar messages [13846.565200] Lustre: Skipped 3 previous similar messages [13850.021794] Lustre: server umount tst250fs-MDT0000 complete [13854.089421] LustreError: 374162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856897 with bad export cookie 31373765888600027 [13854.098580] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13854.105839] LustreError: 374162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13854.514804] Lustre: server umount tst250fs-MDT0001 complete [13868.461498] Lustre: server umount tst250fs-OST0000 complete [13883.067618] Lustre: server umount tst250fs-OST0001 complete [13905.976039] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [13915.534480] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [13949.068255] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 04:43:10 (1788856990) [13951.011856] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13969.979594] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [13982.623985] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [14022.617084] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13680 sec ========= 04:44:23 (1788857063) [14025.450233] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:44:25 (1788857065) === [14029.482495] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:44:30 (1788857070) === [14077.968691] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [14081.743290] Key type lgssc unregistered [14082.168942] LNet: 385820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14082.185389] LNetError: 385820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14082.239697] LNet: Removed LNI 192.168.204.120@tcp [14083.583154] Key type .llcrypt unregistered [14083.589432] Key type ._llcrypt unregistered