[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 492778721 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001032] APIC: Switch to symmetric I/O mode setup [ 0.003375] x2apic enabled [ 0.004018] Switched APIC routing to physical x2apic. [ 0.005024] kvm-guest: setup PV IPIs [ 0.008615] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009034] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010018] pid_max: default: 32768 minimum: 301 [ 0.011164] LSM: Security Framework initializing [ 0.012073] Yama: becoming mindful. [ 0.014057] SELinux: Initializing. [ 0.015118] *** VALIDATE selinux *** [ 0.024458] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029529] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031110] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033057] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034145] *** VALIDATE tmpfs *** [ 0.036475] *** VALIDATE proc *** [ 0.038122] *** VALIDATE cgroup *** [ 0.039014] *** VALIDATE cgroup2 *** [ 0.040319] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.042052] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.043013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.045010] Spectre V2 : User space: Vulnerable [ 0.046017] Speculative Store Bypass: Vulnerable [ 0.049673] debug: unmapping init [mem 0xffffffffbd059000-0xffffffffbd060fff] [ 0.052000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.052954] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.054036] ... version: 2 [ 0.055018] ... bit width: 48 [ 0.056023] ... generic registers: 4 [ 0.057019] ... value mask: 0000ffffffffffff [ 0.058021] ... max period: 00007fffffffffff [ 0.059021] ... fixed-purpose events: 3 [ 0.060018] ... event mask: 000000070000000f [ 0.061381] rcu: Hierarchical SRCU implementation. [ 0.063631] smp: Bringing up secondary CPUs ... [ 0.064646] x86: Booting SMP configuration: [ 0.065033] .... node #0, CPUs: #1 #2 #3 [ 0.070224] smp: Brought up 1 node, 4 CPUs [ 0.072027] smpboot: Max logical packages: 1 [ 0.073013] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.223206] node 0 deferred pages initialised in 147ms [ 0.226190] devtmpfs: initialized [ 0.227232] x86/mm: Memory block size: 128MB [ 0.229936] gcov: version magic: 0x41383552 [ 0.231421] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.232115] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.233315] pinctrl core: initialized pinctrl subsystem [ 0.234263] [ 0.234851] ************************************************************* [ 0.235020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.236023] ** ** [ 0.237019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.238020] ** ** [ 0.239021] ** This means that this kernel is built to expose internal ** [ 0.240019] ** IOMMU data structures, which may compromise security on ** [ 0.241022] ** your system. ** [ 0.242021] ** ** [ 0.243020] ** If you see this message and you are not debugging the ** [ 0.244021] ** kernel, report this immediately to your vendor! ** [ 0.245018] ** ** [ 0.246022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.247023] ************************************************************* [ 0.248797] NET: Registered protocol family 16 [ 0.249564] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.250085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.251101] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.253142] cpuidle: using governor menu [ 0.254844] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.258531] PCI: Using configuration type 1 for base access [ 0.260139] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.270068] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.271041] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.273043] cryptd: max_cpu_qlen set to 1000 [ 0.275328] ACPI: Added _OSI(Module Device) [ 0.276018] ACPI: Added _OSI(Processor Device) [ 0.277018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.278021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.281692] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.284593] ACPI: Interpreter enabled [ 0.285091] ACPI: PM: (supports S0 S3 S4 S5) [ 0.286021] ACPI: Using IOAPIC for interrupt routing [ 0.287173] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.288568] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.298103] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.299069] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.300046] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.301119] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.303000] acpiphp: Slot [2] registered [ 0.305359] acpiphp: Slot [5] registered [ 0.308163] acpiphp: Slot [6] registered [ 0.310155] acpiphp: Slot [7] registered [ 0.312133] acpiphp: Slot [8] registered [ 0.314129] acpiphp: Slot [9] registered [ 0.316532] acpiphp: Slot [10] registered [ 0.318398] acpiphp: Slot [3] registered [ 0.320199] acpiphp: Slot [4] registered [ 0.322114] acpiphp: Slot [11] registered [ 0.324125] acpiphp: Slot [12] registered [ 0.325118] acpiphp: Slot [13] registered [ 0.327106] acpiphp: Slot [14] registered [ 0.328128] acpiphp: Slot [15] registered [ 0.330114] acpiphp: Slot [16] registered [ 0.331178] acpiphp: Slot [17] registered [ 0.333124] acpiphp: Slot [18] registered [ 0.334114] acpiphp: Slot [19] registered [ 0.336094] acpiphp: Slot [20] registered [ 0.337116] acpiphp: Slot [21] registered [ 0.338119] acpiphp: Slot [22] registered [ 0.340129] acpiphp: Slot [23] registered [ 0.342157] acpiphp: Slot [24] registered [ 0.343138] acpiphp: Slot [25] registered [ 0.346200] acpiphp: Slot [26] registered [ 0.348151] acpiphp: Slot [27] registered [ 0.349119] acpiphp: Slot [28] registered [ 0.350111] acpiphp: Slot [29] registered [ 0.352119] acpiphp: Slot [30] registered [ 0.353145] acpiphp: Slot [31] registered [ 0.355117] PCI host bridge to bus 0000:00 [ 0.356029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.359033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.362035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.364032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.368042] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.370044] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.373522] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.377201] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.381666] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.392822] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.398047] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.402037] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.403024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.407028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.410061] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.412903] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.416063] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.421035] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.427023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.440020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.446018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.453022] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.462024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.469023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.502023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.518802] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.526021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.533019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.550019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.563392] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.578022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.587021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.607022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.619628] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.629022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.638022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.660021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.675093] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.682021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.691022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.713024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.726514] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.732014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.737016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.757018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.772252] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.774468] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.777460] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.781522] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.783278] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.788248] iommu: Default domain type: Passthrough [ 0.791463] SCSI subsystem initialized [ 0.793190] ACPI: bus type USB registered [ 0.795137] usbcore: registered new interface driver usbfs [ 0.798120] usbcore: registered new interface driver hub [ 0.801109] usbcore: registered new device driver usb [ 0.803287] pps_core: LinuxPPS API ver. 1 registered [ 0.806016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.810116] PTP clock support registered [ 0.813095] EDAC MC: Ver: 3.0.0 [ 0.815177] PCI: Using ACPI for IRQ routing [ 0.816969] NetLabel: Initializing [ 0.819020] NetLabel: domain hash size = 128 [ 0.821017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.823119] NetLabel: unlabeled traffic allowed by default [ 0.826199] vgaarb: loaded [ 0.827408] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.830021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.837353] clocksource: Switched to clocksource kvm-clock [ 0.955147] VFS: Disk quotas dquot_6.6.0 [ 0.956718] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.959161] *** VALIDATE ramfs *** [ 0.960453] *** VALIDATE hugetlbfs *** [ 0.962020] pnp: PnP ACPI init [ 0.964315] pnp: PnP ACPI: found 6 devices [ 0.987070] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.990221] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.992891] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.995025] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.997338] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.999671] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.002705] NET: Registered protocol family 2 [ 1.005901] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.011222] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.015164] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.020215] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.023957] TCP: Hash tables configured (established 65536 bind 65536) [ 1.027899] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.031195] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.034199] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.037478] NET: Registered protocol family 1 [ 1.040434] RPC: Registered named UNIX socket transport module. [ 1.042352] RPC: Registered udp transport module. [ 1.043685] RPC: Registered tcp transport module. [ 1.045191] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.047065] NET: Registered protocol family 44 [ 1.048490] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.050247] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.052110] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.054227] PCI: CLS 0 bytes, default 64 [ 1.055634] Unpacking initramfs... [ 2.513981] debug: unmapping init [mem 0xffff8baefcc54000-0xffff8baefffbffff] [ 2.521956] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.524655] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.529244] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.013623] Initialise system trusted keyrings [ 3.015486] Key type blacklist registered [ 3.017176] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.025812] zbud: loaded [ 3.028923] *** VALIDATE nfs *** [ 3.030133] *** VALIDATE nfs4 *** [ 3.031879] pstore: using deflate compression [ 3.035234] Platform Keyring initialized [ 3.143637] NET: Registered protocol family 38 [ 3.145879] Key type asymmetric registered [ 3.147658] Asymmetric key parser 'x509' registered [ 3.149494] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.152692] io scheduler mq-deadline registered [ 3.154670] io scheduler kyber registered [ 3.156573] io scheduler bfq registered [ 3.158651] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.161754] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.164214] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.166672] ACPI: Power Button [PWRF] [ 3.172383] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.179906] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.193555] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.200941] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.214760] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.244986] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.275989] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.281023] Non-volatile memory driver v1.3 [ 3.282479] Linux agpgart interface v0.103 [ 3.315225] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.318047] vda: detected capacity change from 0 to 74702848 [ 3.336229] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.339367] vdb: detected capacity change from 0 to 1073741824 [ 3.359552] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.362461] vdc: detected capacity change from 0 to 2621440000 [ 3.378495] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.381894] vdd: detected capacity change from 0 to 2621440000 [ 3.396895] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.400191] vde: detected capacity change from 0 to 4294967296 [ 3.414515] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.417807] vdf: detected capacity change from 0 to 4294967296 [ 3.425217] libphy: Fixed MDIO Bus: probed [ 3.430896] usbcore: registered new interface driver usbserial_generic [ 3.433161] usbserial: USB Serial support registered for generic [ 3.435098] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.438638] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.440476] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.444110] mousedev: PS/2 mouse device common for all mice [ 3.448423] rtc_cmos 00:05: RTC can wake from S4 [ 3.451865] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.452029] rtc_cmos 00:05: registered as rtc0 [ 3.459621] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.462624] intel_pstate: CPU model not supported [ 3.464151] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.471577] hid: raw HID events driver (C) Jiri Kosina [ 3.473852] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.474104] usbcore: registered new interface driver usbhid [ 3.480583] usbhid: USB HID core driver [ 3.482150] drop_monitor: Initializing network drop monitor service [ 3.484249] Initializing XFRM netlink socket [ 3.485880] NET: Registered protocol family 10 [ 3.488782] Segment Routing with IPv6 [ 3.490106] NET: Registered protocol family 17 [ 3.492925] mpls_gso: MPLS GSO support [ 3.499258] RAS: Correctable Errors collector initialized. [ 3.501719] AVX version of gcm_enc/dec engaged. [ 3.503258] AES CTR mode by8 optimization enabled [ 3.581810] sched_clock: Marking stable (3581757815, 0)->(4633850651, -1052092836) [ 3.585263] registered taskstats version 1 [ 3.586715] Loading compiled-in X.509 certificates [ 3.589465] zswap: loaded using pool lzo/zbud [ 3.616436] Key type big_key registered [ 3.630232] Key type encrypted registered [ 3.631927] ima: No TPM chip found, activating TPM-bypass! [ 3.633936] ima: Allocated hash algorithm: sha1 [ 3.635683] ima: No architecture policies found [ 3.637569] evm: Initialising EVM extended attributes: [ 3.639853] evm: security.selinux [ 3.641420] evm: security.ima [ 3.642646] evm: security.capability [ 3.644331] evm: HMAC attrs: 0x1 [ 3.648164] rtc_cmos 00:05: setting system clock to 2026-08-19 00:14:53 UTC (1787098493) [ 3.654066] debug: unmapping init [mem 0xffffffffbe003000-0xffffffffbe1fffff] [ 3.656736] debug: unmapping init [mem 0xffffffffbcd82000-0xffffffffbd058fff] [ 3.666099] Write protecting the kernel read-only data: 28672k [ 3.669493] debug: unmapping init [mem 0xffffffffbb403000-0xffffffffbb5fffff] [ 3.673371] debug: unmapping init [mem 0xffffffffbbd14000-0xffffffffbbdfffff] [ 3.707388] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.717526] systemd[1]: Detected virtualization kvm. [ 3.719541] systemd[1]: Detected architecture x86-64. [ 3.721743] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.748804] systemd[1]: No hostname configured. [ 3.750541] systemd[1]: Set hostname to . [ 3.752771] random: systemd: uninitialized urandom read (16 bytes read) [ 3.755353] systemd[1]: Initializing machine ID from random generator. [ 3.905772] random: systemd: uninitialized urandom read (16 bytes read) [ 3.908487] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.912383] random: systemd: uninitialized urandom read (16 bytes read) [ 3.914837] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.920382] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.591646] device-mapper: uevent: version 1.0.3 [ 4.593966] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.329353] random: fast init done [ 5.364241] virtio_net virtio0 ens2: renamed from eth0 [ 5.419077] scsi host0: ata_piix [ 5.435977] scsi host1: ata_piix [ 5.437784] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.441145] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.893364] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.152110] random: crng init done [ 10.153427] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.616400] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.764561] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.027790] SELinux: Disabled at runtime. [ 12.090702] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.100134] systemd[1]: Detected virtualization kvm. [ 12.102260] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.608542] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.612613] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.617432] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.621567] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.625199] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.633809] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.646376] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ 12.828484] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.168745] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.496764] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.575388] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.726603] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.747897] EDAC sbridge: Ver: 1.1.2 [ 15.405242] Key type dns_resolver registered [ 15.711080] NFS: Registering the id_resolver key type [ 15.713270] Key type id_resolver registered [ 15.715170] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg430-server login: [ 42.782883] libcfs: loading out-of-tree module taints kernel. [ 42.810172] Key type ._llcrypt registered [ 42.812869] Key type .llcrypt registered [ 42.871601] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 54.316127] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 55.872412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 55.888133] alg: No test for adler32 (adler32-zlib) [ 57.382735] Lustre: Lustre: Build Version: 2.17.57_1_g147bb23 [ 58.733969] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 60.608194] Key type lgssc registered [ 62.489980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 78.380355] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 105.283066] hrtimer: interrupt took 2214814 ns [ 122.063748] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 136.458263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 136.533989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 137.860373] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 137.896712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 137.999625] Lustre: lustre-MDT0000: new disk, initializing [ 138.076646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 138.118545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 143.032500] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 157.319362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 157.462428] Lustre: 6488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 157.503574] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 157.507990] Lustre: Skipped 1 previous similar message [ 157.621573] Lustre: lustre-MDT0001: new disk, initializing [ 157.727522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 157.761646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 157.776406] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 163.175750] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.788535] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 179.363584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 179.602940] Lustre: lustre-OST0000: new disk, initializing [ 179.608323] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 179.614435] Lustre: 8427:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 179.699830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 183.486760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 183.501944] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 183.591512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 186.466549] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.988221] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 204.068668] Lustre: lustre-OST0001: new disk, initializing [ 204.071052] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 204.078482] Lustre: 9499:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 204.131847] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 211.719466] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.544418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 212.558472] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 212.645863] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 226.441571] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 233.898706] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 240.876774] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing check_logdir /tmp/testlogs/ [ 246.377499] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing yml_node [ 250.733628] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 253.407399] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 256.507376] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 258.407651] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Aug 18 20:19:06 EDT 2026 [ 277.250472] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 278.974854] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 295.913826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 295.926638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 295.946924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 300.010310] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.011011] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 300.025063] Lustre: Skipped 2 previous similar messages [ 300.040171] Lustre: Skipped 3 previous similar messages [ 302.378061] Lustre: server umount lustre-MDT0000 complete [ 306.220123] LustreError: 6480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787098796 with bad export cookie 10674946432913743446 [ 306.223839] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 306.231063] LustreError: 6480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 310.246913] LustreError: 6495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.249357] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.257192] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 310.265617] LustreError: 6495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 310.283122] Lustre: Skipped 1 previous similar message [ 311.265070] LustreError: 8891:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 311.278043] LustreError: 8891:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 312.650665] Lustre: server umount lustre-MDT0001 complete [ 317.247685] Lustre: server umount lustre-OST0000 complete [ 321.722894] Lustre: server umount lustre-OST0001 complete [ 328.352403] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 338.279789] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 375.220865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 375.485394] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 375.525532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 375.621434] Lustre: lustre-MDT0000: new disk, initializing [ 375.724733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 375.742710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 380.734749] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 390.789501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 390.893857] Lustre: 15055:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 390.956863] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 390.963245] Lustre: Skipped 1 previous similar message [ 391.073422] Lustre: lustre-MDT0001: new disk, initializing [ 391.166673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 391.215694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 391.234599] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 395.337408] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 404.561679] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 410.306161] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 417.342517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 417.635423] Lustre: lustre-OST0000: new disk, initializing [ 417.647051] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 417.658629] Lustre: 17024:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 417.793383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 418.779536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 418.796628] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 418.949337] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 423.748557] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 432.856173] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 436.743649] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 436.990550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 441.354875] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 441.532158] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 444.391906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 444.392691] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 444.393104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 444.393109] Lustre: Skipped 3 previous similar messages [ 444.419025] Lustre: Skipped 1 previous similar message [ 449.507570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 449.521120] Lustre: Skipped 2 previous similar messages [ 449.906102] Lustre: server umount lustre-OST0000 complete [ 454.631411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 459.745359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 459.753890] Lustre: Skipped 3 previous similar messages [ 460.456179] Lustre: server umount lustre-MDT0000 complete [ 464.429715] LustreError: 15045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787098954 with bad export cookie 10674946432913744979 [ 464.431179] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 464.445808] LustreError: 15045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 465.181776] Lustre: server umount lustre-MDT0001 complete [ 471.902948] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 20:22:39 (1787098959) [ 491.308475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.919734] LustreError: 19898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 492.045380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 497.130855] LustreError: 19899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.151380] LustreError: 19899:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 497.269532] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 501.219824] LustreError: 19898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 505.901436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 510.311992] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 518.683752] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 524.916838] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 527.324492] Lustre: Failing over lustre-MDT0000 [ 527.354262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 527.360967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 527.374286] Lustre: Skipped 2 previous similar messages [ 527.391345] Lustre: Skipped 1 previous similar message [ 527.629841] Lustre: server umount lustre-MDT0000 complete [ 530.471298] LustreError: 19899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.492262] LustreError: 19899:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 546.402021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 546.551477] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 550.894990] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 550.907305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 550.931298] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 551.073954] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 561.902434] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 566.242835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 566.249436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 566.267845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 571.871463] Lustre: server umount lustre-MDT0000 complete [ 572.389102] LustreError: 19898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.413947] LustreError: 19898:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 575.006032] LustreError: 19878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787099064 with bad export cookie 10674946432913746393 [ 575.014713] LustreError: 19878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 575.014752] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 575.241399] Lustre: server umount lustre-MDT0001 complete [ 583.068386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.575900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 583.593250] Lustre: Skipped 2 previous similar messages [ 587.668977] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 596.581936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.542207] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 609.215705] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 615.286815] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 617.783281] Lustre: Failing over lustre-MDT0000 [ 618.020762] Lustre: server umount lustre-MDT0000 complete [ 618.465860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 618.466856] LustreError: 23234:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 618.494358] Lustre: Skipped 2 previous similar messages [ 618.516490] LustreError: 23234:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 634.728496] Lustre: 3622:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787099108/real 1787099108] req@ffff8bae4267fb80 x1873908647257728/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787099124 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 634.795270] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 638.528107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 645.450719] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 650.724790] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 650.735544] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 650.739101] Lustre: Skipped 1 previous similar message [ 650.762054] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 650.994554] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 660.430981] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 666.082721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 666.083838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 666.100000] Lustre: Skipped 2 previous similar messages [ 670.576609] Lustre: server umount lustre-MDT0000 complete [ 674.082840] LustreError: 23218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787099163 with bad export cookie 10674946432913747331 [ 674.109158] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 674.112623] LustreError: 23218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 674.454109] Lustre: server umount lustre-MDT0001 complete [ 682.957590] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 20:26:10 (1787099170) [ 701.064333] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 712.157678] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 755.041384] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 768.166227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 768.489545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 768.546390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 768.624879] Lustre: lustre-MDT0000: new disk, initializing [ 768.757881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 768.765252] Lustre: Skipped 2 previous similar messages [ 768.787453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 774.513843] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 785.452682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 785.572377] Lustre: 29611:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 785.881858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 790.333905] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 800.683433] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 806.461468] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 813.313039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 813.683350] Lustre: 31581:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 815.305780] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 815.344546] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 815.394523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 815.412761] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 821.603437] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 833.698791] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 840.370296] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 871.908792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 871.926362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 871.946762] Lustre: Skipped 2 previous similar messages [ 871.956753] LustreError: Skipped 1 previous similar message [ 871.966402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 871.982534] Lustre: Skipped 3 previous similar messages [ 874.527712] Lustre: server umount lustre-OST0000 complete [ 886.538566] Lustre: server umount lustre-MDT0000 complete [ 887.294939] LustreError: 29625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 887.313570] LustreError: 29625:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 890.866983] LustreError: 30417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787099380 with bad export cookie 10674946432913747737 [ 890.873849] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 890.874556] LustreError: 30417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 891.389020] Lustre: server umount lustre-MDT0001 complete [ 904.822805] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 908.090450] Key type lgssc unregistered [ 908.479416] LNet: 33566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.485758] LNetError: 33566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.513383] LNet: Removed LNI 192.168.204.130@tcp [ 909.619269] Key type .llcrypt unregistered [ 909.621734] Key type ._llcrypt unregistered [ 940.119355] Key type ._llcrypt registered [ 940.124172] Key type .llcrypt registered [ 940.386339] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 942.745171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 942.816171] alg: No test for adler32 (adler32-zlib) [ 944.103277] Lustre: Lustre: Build Version: 2.17.57_1_g147bb23 [ 944.299379] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 945.968210] Key type lgssc registered [ 948.045784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 956.698907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 956.729719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 958.245138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 962.218313] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 968.674500] LustreError: 34982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 968.697799] LustreError: 34982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 971.226161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 971.818155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 976.963124] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 985.590231] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 992.494103] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1000.614143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1001.206616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1003.291554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1007.188015] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1014.445971] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1016.557756] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1030.675671] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 20:31:58 (1787099518) [ 1032.771253] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1035.165177] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1042.505811] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 20:32:09 (1787099529) [ 1051.936828] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 20:32:19 (1787099539) [ 1052.946884] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1062.762719] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1062.772730] Lustre: Skipped 2 previous similar messages [ 1065.099857] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 20:32:32 (1787099552) [ 1067.922918] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1067.933319] Lustre: Skipped 2 previous similar messages [ 1076.703833] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1076.716879] Lustre: Skipped 1 previous similar message [ 1093.049527] Lustre: Disabling parameter general.osc.*.max_pages_per_rpc= in log params [ 1093.055338] Lustre: Skipped 8 previous similar messages [ 1094.937496] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 20:33:02 (1787099582) [ 1097.220829] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1100.555418] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1108.621309] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 20:33:16 (1787099596) [ 1126.858344] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 20:33:34 (1787099614) [ 1144.802160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1144.812318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.822370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1144.826194] Lustre: Skipped 1 previous similar message [ 1144.837490] Lustre: Skipped 1 previous similar message [ 1148.921470] Lustre: server umount lustre-OST0000 complete [ 1155.044264] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1155.071699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1155.102422] Lustre: Skipped 1 previous similar message [ 1160.161703] LustreError: 34977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1160.233307] LustreError: 34977:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1160.264221] Lustre: server umount lustre-MDT0000 complete [ 1164.267249] LustreError: 34982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.318841] LustreError: 34982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1166.432950] LustreError: 34962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787099656 with bad export cookie 16712250521114855526 [ 1166.455984] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1166.462358] LustreError: 34962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1167.236482] Lustre: server umount lustre-MDT0001 complete [ 1184.181157] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 1186.916028] Key type lgssc unregistered [ 1187.368749] LNet: 42855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1187.381715] LNetError: 42855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1187.417592] LNet: Removed LNI 192.168.204.130@tcp [ 1189.006187] Key type .llcrypt unregistered [ 1189.010762] Key type ._llcrypt unregistered [ 1209.338509] Key type ._llcrypt registered [ 1209.341989] Key type .llcrypt registered [ 1209.440985] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 1226.179825] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 1227.633121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1227.692209] alg: No test for adler32 (adler32-zlib) [ 1228.781621] Lustre: Lustre: Build Version: 2.17.57_1_g147bb23 [ 1228.989653] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 1230.704312] Key type lgssc registered [ 1231.951872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1275.315421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1275.361968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1276.710420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1276.759722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1276.857879] Lustre: lustre-MDT0000: new disk, initializing [ 1276.930038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1276.950146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1282.504542] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1294.108575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1294.269654] Lustre: 46008:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1294.329684] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1294.333977] Lustre: Skipped 1 previous similar message [ 1294.388917] Lustre: lustre-MDT0001: new disk, initializing [ 1294.456086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1294.484244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1294.510915] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1298.990664] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1309.227474] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1315.705062] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1323.960224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1324.209908] Lustre: lustre-OST0000: new disk, initializing [ 1324.216635] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1324.221930] Lustre: 47980:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1324.322642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1330.479723] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1331.280565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1331.291932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1331.446148] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1342.974089] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1350.234684] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1350.587820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1356.870466] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1357.165763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1362.402438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1362.405918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.420245] LustreError: Skipped 1 previous similar message [ 1362.426784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1362.465559] Lustre: Skipped 1 previous similar message [ 1365.827775] Lustre: server umount lustre-OST0000 complete [ 1371.107688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1371.119123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1371.139761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1371.144843] Lustre: Skipped 1 previous similar message [ 1372.643430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1372.653707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1376.000246] Lustre: server umount lustre-MDT0000 complete [ 1379.539561] LustreError: 46803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787099869 with bad export cookie 9970851394989478629 [ 1379.547788] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1379.551820] LustreError: 46803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1379.852898] Lustre: server umount lustre-MDT0001 complete [ 1389.660786] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 20:37:57 (1787099877) [ 1396.873427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1397.518070] LustreError: 50443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1397.534760] LustreError: 50443:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1397.608730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1401.782225] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1402.855856] LustreError: 50444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1407.973663] LustreError: 50443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1411.010944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1411.453917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1417.589161] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1428.199351] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1437.160828] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1444.355261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1444.622678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1450.756761] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1459.286099] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1475.357766] LustreError: 52995:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1476.591621] Lustre: Modifying parameter general.timeout=1 in log params [ 1477.653497] Lustre: Modifying parameter general.timeout=2 in log params [ 1480.801905] Lustre: Modifying parameter general.timeout=5 in log params [ 1480.805349] Lustre: Skipped 2 previous similar messages [ 1485.217448] Lustre: Modifying parameter general.timeout=9 in log params [ 1485.220765] Lustre: Skipped 3 previous similar messages [ 1493.423604] Lustre: Modifying parameter general.timeout=16 in log params [ 1493.430588] Lustre: Skipped 6 previous similar messages [ 1509.769560] Lustre: Modifying parameter general.timeout=30 in log params [ 1509.774327] Lustre: Skipped 13 previous similar messages [ 1542.317542] Lustre: Modifying parameter general.timeout=60 in log params [ 1542.322142] Lustre: Skipped 29 previous similar messages [ 1607.037333] Lustre: Modifying parameter general.timeout=122 in log params [ 1607.039742] Lustre: Skipped 61 previous similar messages [ 1626.343262] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 20:41:53 (1787100113) [ 1629.534572] Lustre: 59949:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1631.717823] Lustre: 60049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1634.075720] Lustre: 60159:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1638.672766] Lustre: 60358:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1638.694468] Lustre: 60358:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1676.889640] Lustre: 61829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1676.898499] Lustre: 61829:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1676.922938] LustreError: 61829:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1682.311280] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 20:42:48 (1787100168) [ 1698.273958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1698.288407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1698.307559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1698.317920] Lustre: Skipped 1 previous similar message [ 1704.654529] Lustre: server umount lustre-MDT0000 complete [ 1706.466876] LustreError: 51171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1706.496943] LustreError: 51171:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1711.112319] LustreError: 50424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787100200 with bad export cookie 9970851394989479511 [ 1711.125221] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1711.150272] LustreError: 50424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1711.591425] LustreError: 50439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1711.633406] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1711.658402] Lustre: Skipped 2 previous similar messages [ 1711.934794] Lustre: server umount lustre-MDT0001 complete [ 1719.166972] Lustre: server umount lustre-OST0000 complete [ 1757.828740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1758.273145] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1758.346539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1758.376760] Lustre: Skipped 30 previous similar messages [ 1758.688907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1764.936912] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1776.298716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1776.413063] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1776.454770] Lustre: 64364:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1783.049322] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1792.738698] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1800.483752] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1808.198279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1808.402236] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1808.726377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1808.734770] Lustre: Skipped 1 previous similar message [ 1815.797662] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1822.748561] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1824.768482] Lustre: 66932:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1837.538299] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1837.541765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1837.548123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1837.551190] Lustre: Skipped 3 previous similar messages [ 1840.730649] Lustre: server umount lustre-OST0000 complete [ 1848.290440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1848.300262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1848.312513] LustreError: Skipped 1 previous similar message [ 1848.335850] Lustre: Skipped 3 previous similar messages [ 1853.294098] Lustre: server umount lustre-MDT0000 complete [ 1853.415877] LustreError: 64370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1853.436410] LustreError: 64370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1859.708759] LustreError: 64354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787100349 with bad export cookie 9970851394989482724 [ 1859.725791] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1859.741926] LustreError: 64354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1860.710833] Lustre: server umount lustre-MDT0001 complete [ 1875.596734] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 1879.443677] Key type lgssc unregistered [ 1879.884911] LNet: 68149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1879.891931] LNetError: 68149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1879.908929] LNet: Removed LNI 192.168.204.130@tcp [ 1880.830939] Key type .llcrypt unregistered [ 1880.834352] Key type ._llcrypt unregistered [ 1912.708401] Key type ._llcrypt registered [ 1912.717099] Key type .llcrypt registered [ 1912.868712] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 1914.540246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1914.583968] alg: No test for adler32 (adler32-zlib) [ 1915.722334] Lustre: Lustre: Build Version: 2.17.57_1_g147bb23 [ 1916.025779] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 1917.672198] Key type lgssc registered [ 1919.838499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1930.694607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1930.715528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1932.410166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1937.268382] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1943.019371] LustreError: 69566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1943.052964] LustreError: 69566:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1946.498174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1947.136845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1952.645552] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1960.849237] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1966.915718] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1974.692572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1975.179648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1982.900850] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 1991.909670] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2002.460478] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2017.258265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2017.260438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2017.276996] Lustre: Skipped 2 previous similar messages [ 2017.303389] Lustre: Skipped 2 previous similar messages [ 2022.408207] LustreError: 69562:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2022.578481] Lustre: server umount lustre-MDT0000 complete [ 2027.255859] LustreError: 69547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787100517 with bad export cookie 5205124564100239643 [ 2027.271511] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2027.273082] LustreError: 69547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2027.608473] Lustre: server umount lustre-MDT0001 complete [ 2043.654833] Lustre: server umount lustre-OST0000 complete [ 2074.652934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2075.027897] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2075.068211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2075.392877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2080.860881] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2090.219203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2090.302441] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2090.374973] Lustre: 74442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2091.104892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2096.442594] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2107.051489] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2113.591481] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2124.271714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2124.594836] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2125.080941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2136.427948] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2145.887944] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2148.357677] Lustre: 77022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2150.817741] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2159.073789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2159.086229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2159.094680] LustreError: Skipped 1 previous similar message [ 2159.120478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2159.128460] Lustre: Skipped 1 previous similar message [ 2164.193730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2164.211515] Lustre: Skipped 1 previous similar message [ 2164.762227] Lustre: server umount lustre-OST0000 complete [ 2172.898110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2172.914763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2172.943357] Lustre: Skipped 1 previous similar message [ 2172.968258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2176.145540] Lustre: server umount lustre-MDT0000 complete [ 2179.554380] LustreError: 75653:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2179.580652] LustreError: 75653:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2180.839524] LustreError: 74437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787100670 with bad export cookie 5205124564100240490 [ 2180.840813] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2180.861192] LustreError: 74437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2181.289273] Lustre: server umount lustre-MDT0001 complete [ 2200.380828] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 2206.280444] Key type lgssc unregistered [ 2206.845178] LNet: 78258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2206.863369] LNetError: 78258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2206.888883] LNet: Removed LNI 192.168.204.130@tcp [ 2208.540071] Key type .llcrypt unregistered [ 2208.541724] Key type ._llcrypt unregistered [ 2239.697741] Key type ._llcrypt registered [ 2239.702279] Key type .llcrypt registered [ 2239.821576] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 2241.154181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2241.314073] alg: No test for adler32 (adler32-zlib) [ 2242.531566] Lustre: Lustre: Build Version: 2.17.57_1_g147bb23 [ 2242.889197] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 2244.545571] Key type lgssc registered [ 2246.246810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2257.062565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2257.100758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2258.803325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2263.550960] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2269.156885] LustreError: 79675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2269.210559] LustreError: 79675:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2274.277077] LustreError: 79676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2275.823080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2277.100833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2284.425281] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2295.294168] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2303.443736] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2315.911662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2316.367769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2328.441902] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2340.588833] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2348.320971] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2349.433772] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2350.558557] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2352.646876] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2352.653667] Lustre: Skipped 1 previous similar message [ 2357.532421] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2357.536925] Lustre: Skipped 4 previous similar messages [ 2366.054780] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2366.059158] Lustre: Skipped 7 previous similar messages [ 2383.012688] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2383.015413] Lustre: Skipped 14 previous similar messages [ 2415.788750] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2415.797162] Lustre: Skipped 22 previous similar messages [ 2466.187511] Lustre: 87790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2468.860958] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 20:55:56 (1787100956) [ 2470.419809] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2471.893549] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 20:55:59 (1787100959) [ 2475.500286] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2475.512666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2475.536510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2476.518959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.519141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2476.539019] Lustre: Skipped 1 previous similar message [ 2481.661715] Lustre: server umount lustre-OST0000 complete [ 2486.242418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2486.254742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2486.272831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2490.897184] Lustre: server umount lustre-MDT0000 complete [ 2491.874244] LustreError: 79676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2491.890218] LustreError: 79676:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2493.748787] LustreError: 79655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787100983 with bad export cookie 10605575204294915710 [ 2493.749924] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2493.757899] LustreError: 79655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2494.034203] Lustre: server umount lustre-MDT0001 complete [ 2503.040864] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 2505.303027] Key type lgssc unregistered [ 2505.520900] LNet: 89039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2505.525297] LNetError: 89039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2505.536038] LNet: Removed LNI 192.168.204.130@tcp [ 2506.223766] Key type .llcrypt unregistered [ 2506.229434] Key type ._llcrypt unregistered [ 2510.033872] Key type ._llcrypt registered [ 2510.036187] Key type .llcrypt registered [ 2510.134439] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2513.870395] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules [ 2514.661613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2514.733755] alg: No test for adler32 (adler32-zlib) [ 2515.753725] Lustre: Lustre: Build Version: 2.17.57_1_g147bb23 [ 2515.928440] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 2517.632348] Key type lgssc registered [ 2518.591692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2519.661748] LustreError: 90248:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2533.376446] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 2540.039400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2541.299741] LustreError: 90248:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2541.350612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2542.758188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2546.215975] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2552.290185] LustreError: 91272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.307873] LustreError: 91272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2552.953751] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 20:57:20 (1787101040) [ 2556.438921] LustreError: 91303:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2556.445481] LustreError: 91303:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2556.450663] LustreError: 91303:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 2556.646588] Lustre: server umount lustre-MDT0000 complete [ 2566.074078] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 2568.076495] Key type lgssc unregistered [ 2568.314678] LNet: 92801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2568.319165] LNetError: 92801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2568.335635] LNet: Removed LNI 192.168.204.130@tcp [ 2568.924775] Key type .llcrypt unregistered [ 2568.927285] Key type ._llcrypt unregistered [ 2581.103814] Key type ._llcrypt registered [ 2581.105856] Key type .llcrypt registered [ 2581.181749] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 2581.890149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2582.015652] alg: No test for adler32 (adler32-zlib) [ 2583.031553] Lustre: Lustre: Build Version: 2.17.57_1_g147bb23 [ 2583.224347] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 2584.880276] Key type lgssc registered [ 2585.710943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2591.264871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2591.277776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2592.598170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2595.827779] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2601.528745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2601.992544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2604.790955] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2609.394152] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2612.796573] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2616.957946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2617.362922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2621.298936] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2625.701458] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2656.139441] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 20:59:04 (1787101144) [ 2657.022752] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2658.000154] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 20:59:06 (1787101146) [ 2662.371614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2662.374405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2662.379568] Lustre: Skipped 1 previous similar message [ 2662.386398] Lustre: Skipped 2 previous similar messages [ 2666.182600] Lustre: server umount lustre-MDT0000 complete [ 2667.493127] LustreError: 93994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2667.509849] LustreError: 93994:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2668.128817] LustreError: 94740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787101157 with bad export cookie 12714830773891104593 [ 2668.135555] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2668.136700] LustreError: 94740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2668.320695] Lustre: server umount lustre-MDT0001 complete [ 2680.814625] Lustre: server umount lustre-OST0000 complete [ 2687.207547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2687.461941] LustreError: 98267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2687.560146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2689.829275] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2692.577201] LustreError: 98268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2694.394295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2694.742373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2696.708474] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2700.213175] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2702.922558] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2709.273334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2709.372543] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2709.376841] LustreError: 98256:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2709.393484] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2709.400077] LustreError: 100354:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2709.404902] LustreError: 100354:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2709.409147] LustreError: 100354:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2709.457093] Lustre: server umount lustre-OST0000 complete [ 2709.460185] LustreError: 100354:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2713.655056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2713.765885] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2713.769869] LustreError: 98255:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2713.785252] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2713.789698] LustreError: 100820:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2713.793563] LustreError: 100820:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2713.796931] LustreError: 100820:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2713.834859] Lustre: server umount lustre-OST0000 complete [ 2713.838409] LustreError: 100820:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2718.650728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2718.757222] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2718.761624] Lustre: Found index 0 for lustre-OST0000, updating log [ 2718.767043] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2718.808781] Lustre: lustre-OST0000: new disk, initializing [ 2718.811746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2718.815233] Lustre: 101352:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2718.947829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2721.800986] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2723.728147] Lustre: Failing over lustre-OST0000 [ 2723.840866] Lustre: server umount lustre-OST0000 complete [ 2730.465366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2730.470097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2730.476389] Lustre: Skipped 1 previous similar message [ 2730.479676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2731.892585] Lustre: server umount lustre-MDT0000 complete [ 2733.792604] LustreError: 98249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787101223 with bad export cookie 12714830773891107337 [ 2733.799345] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2733.800573] LustreError: 98249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2743.343803] Lustre: server umount lustre-MDT0001 complete [ 2747.543895] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 21:00:36 (1787101236) [ 2755.179712] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 2760.100534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2760.284606] LustreError: 103694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2760.336546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2762.147844] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2765.794617] LustreError: 103695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.202463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2768.311874] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2769.578025] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2772.786210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2773.156253] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2775.972410] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2776.233571] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2776.254094] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2776.254346] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2779.930085] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2779.972251] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2780.021123] Lustre: lustre-OST0001: new disk, initializing [ 2780.023615] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2780.028898] Lustre: 106025:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2782.678077] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2783.439871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2783.444863] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2783.490099] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2786.637569] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2791.920149] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2794.215318] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.30@tcp (stopping) [ 2794.219781] Lustre: Skipped 1 previous similar message [ 2796.003855] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2796.011284] Lustre: Skipped 3 previous similar messages [ 2799.012135] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2799.015784] Lustre: Skipped 5 previous similar messages [ 2804.196364] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2804.199573] Lustre: Skipped 3 previous similar messages [ 2808.288132] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2808.364835] Lustre: server umount lustre-MDT0001 complete [ 2809.314771] LustreError: 103691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2809.322364] LustreError: 103691:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2812.881153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2812.943700] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2812.948788] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2812.976295] Lustre: 103682:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2813.350366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2813.353409] Lustre: Skipped 3 previous similar messages [ 2814.121548] LustreError: 107616:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2814.126234] LustreError: 107616:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2814.131508] LustreError: 103686:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2814.188631] LustreError: 107627:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.130@tcp: cfg command failed: rc = -17 [ 2814.193788] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2814.193788] [ 2815.308445] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2818.537338] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2819.586511] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 21:01:48 (1787101308) [ 2827.498536] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 2835.747232] Lustre: 109634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2847.317596] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2848.988772] Lustre: 110767:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2910.689372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2910.689442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2910.695374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2910.701456] Lustre: Skipped 4 previous similar messages [ 2912.282373] Lustre: server umount lustre-MDT0000 complete [ 2913.663776] LustreError: 106024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787101403 with bad export cookie 12714830773891108268 [ 2913.665495] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2913.668454] LustreError: 106024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2913.776404] Lustre: server umount lustre-MDT0001 complete [ 2926.020935] Lustre: server umount lustre-OST0000 complete [ 2937.771954] Lustre: server umount lustre-OST0001 complete [ 2940.146029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2944.031917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2947.411192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2951.047043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2958.442791] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 2962.534230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2962.544644] Lustre: lustre-MDT0000: reset Object Index mappings [ 2962.731856] LustreError: 114714:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2962.738685] LustreError: 114714:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2962.783415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2964.403860] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2968.053018] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2968.061159] Lustre: lustre-MDT0001: reset Object Index mappings [ 2969.917466] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2971.096289] Lustre: 115860:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2973.718508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2976.270258] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2979.178771] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 2979.588200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2982.029990] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 2984.938129] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2988.017990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 2988.018781] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2990.802313] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2992.072923] Lustre: 117737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3008.866665] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 21:04:57 (1787101497) [ 3039.204831] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3039.208071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3039.210203] Lustre: Skipped 6 previous similar messages [ 3039.213104] Lustre: Skipped 2 previous similar messages [ 3044.202473] Lustre: server umount lustre-MDT0000 complete [ 3044.322924] LustreError: 114709:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3044.329122] LustreError: 114709:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3045.540439] LustreError: 115463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787101535 with bad export cookie 12714830773891155063 [ 3045.544816] LustreError: 115463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3045.545427] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3049.443172] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3049.446772] Lustre: Skipped 1 previous similar message [ 3051.855863] Lustre: server umount lustre-MDT0001 complete [ 3059.392503] Lustre: server umount lustre-OST0000 complete [ 3066.782296] Lustre: server umount lustre-OST0001 complete [ 3069.009773] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 3072.425711] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 3085.447222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3085.531725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3085.543510] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3085.583334] Lustre: lustre-MDT0000: new disk, initializing [ 3085.611060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3087.160936] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3090.209871] Lustre: server umount lustre-MDT0000 complete [ 3093.545807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3093.648668] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3093.654506] Lustre: 121992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3093.660308] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3093.732384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3093.734866] Lustre: Skipped 4 previous similar messages [ 3095.179368] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3096.982210] Lustre: server umount lustre-MDT0000 complete [ 3100.222223] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 21:06:28 (1787101588) [ 3100.874107] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3101.548121] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 21:06:30 (1787101590) [ 3108.069595] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 3111.706212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3113.362245] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3116.364096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3116.407906] Lustre: 124040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3116.434154] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3116.437015] Lustre: Skipped 1 previous similar message [ 3116.473030] Lustre: lustre-MDT0001: new disk, initializing [ 3116.520744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3116.524861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3117.927657] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3120.266656] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3122.645849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3122.753205] Lustre: lustre-OST0000: new disk, initializing [ 3122.755388] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3122.758705] Lustre: 125618:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3123.316925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3123.321482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3123.356992] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3124.826755] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3129.224982] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3129.287732] Lustre: 126487:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3130.764654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3130.768029] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3130.802555] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3131.413000] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3135.860904] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3137.014269] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3292.194872] Lustre: 132864:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3292.199064] Lustre: 132864:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3292.202366] Lustre: 132864:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3292.206025] Lustre: 132864:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3292.210197] Lustre: 132864:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3292.213531] Lustre: 132864:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3374.967081] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 21:11:03 (1787101863) [ 3386.853716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3386.854811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3386.858706] Lustre: Skipped 2 previous similar messages [ 3386.860743] Lustre: Skipped 5 previous similar messages [ 3392.636029] Lustre: server umount lustre-MDT0000 complete [ 3393.912416] LustreError: 132877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787101883 with bad export cookie 12714830773891209873 [ 3393.916487] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3393.917446] LustreError: 132877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3394.059839] Lustre: server umount lustre-MDT0001 complete [ 3411.386547] Lustre: server umount lustre-OST0000 complete [ 3412.448183] Lustre: 93309:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787101886/real 1787101886] req@ffff8bae49d95180 x1873911301335424/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787101902 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3418.783075] Lustre: server umount lustre-OST0001 complete [ 3421.001574] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 3424.229594] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 3436.171190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3436.253105] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3436.264728] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3436.267105] Lustre: Skipped 1 previous similar message [ 3436.302532] Lustre: lustre-MDT0000: new disk, initializing [ 3436.304973] Lustre: Skipped 1 previous similar message [ 3436.325170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3436.328130] Lustre: Skipped 4 previous similar messages [ 3436.334218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3437.735250] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3441.849619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3441.888492] Lustre: 135876:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3441.980481] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3443.296218] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3446.878592] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3448.666297] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3450.811345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3450.907543] Lustre: 137845:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3452.761768] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3452.918048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3452.920779] Lustre: Skipped 1 previous similar message [ 3452.922760] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3452.963482] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3456.413393] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3459.490026] Lustre: lustre-MDD0000: changelog on [ 3460.346228] Lustre: lustre-MDD0001: changelog on [ 3794.881765] Lustre: lustre-MDD0001: changelog off [ 3795.594657] Lustre: 140335:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 3795.955437] Lustre: lustre-MDD0000: changelog off [ 3801.058661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3801.059931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3801.064773] LustreError: Skipped 1 previous similar message [ 3801.069777] Lustre: Skipped 2 previous similar messages [ 3801.072512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3801.076295] Lustre: Skipped 6 previous similar messages [ 3812.320133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3812.391256] Lustre: server umount lustre-OST0000 complete [ 3815.904902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3828.192109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3828.301607] Lustre: server umount lustre-MDT0000 complete [ 3829.596502] LustreError: 135911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787102319 with bad export cookie 12714830773891976177 [ 3829.600032] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3829.600885] LustreError: 135911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3829.736349] Lustre: server umount lustre-MDT0001 complete [ 3833.502846] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 3834.597755] Key type lgssc unregistered [ 3834.733453] LNet: 146152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3834.737308] LNetError: 146152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3834.746306] LNet: Removed LNI 192.168.204.130@tcp [ 3835.103205] Key type .llcrypt unregistered [ 3835.104458] Key type ._llcrypt unregistered [ 3841.473991] Key type ._llcrypt registered [ 3841.474902] Key type .llcrypt registered [ 3841.509647] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 3847.737317] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 3848.074665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3848.115643] alg: No test for adler32 (adler32-zlib) [ 3848.977442] Lustre: Lustre: Build Version: 2.17.57_1_g147bb23 [ 3849.067299] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 3850.648190] Key type lgssc registered [ 3851.000656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3862.818577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3862.825758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3863.920171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3863.931076] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3863.968431] Lustre: lustre-MDT0000: new disk, initializing [ 3863.991154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3863.997096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3865.402389] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3869.646195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3869.690122] Lustre: 149325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3869.712381] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3869.714929] Lustre: Skipped 1 previous similar message [ 3869.752911] Lustre: lustre-MDT0001: new disk, initializing [ 3869.787447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3869.801343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3869.805585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3871.143112] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3874.908041] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3876.706528] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3879.040344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3879.153388] Lustre: lustre-OST0000: new disk, initializing [ 3879.155323] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3879.158847] Lustre: 151294:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3879.186890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3879.342192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3879.347027] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3879.376626] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3881.188454] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3884.810316] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3887.039876] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_module obdecho/obdecho [ 3887.649989] LustreError: 152297:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 3887.652836] LustreError: 152297:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 3888.257886] LustreError: 152393:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 3894.755764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3894.757858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3894.759563] Lustre: Skipped 2 previous similar messages [ 3894.762771] Lustre: Skipped 2 previous similar messages [ 3897.496757] Lustre: server umount lustre-MDT0000 complete [ 3898.797344] LustreError: 150121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787102388 with bad export cookie 14410134653486835972 [ 3898.799934] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3898.802824] LustreError: 150121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3898.920220] Lustre: server umount lustre-MDT0001 complete [ 3910.491246] Lustre: server umount lustre-OST0000 complete [ 3917.408020] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 3920.953083] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 3932.657397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3932.755977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3932.766372] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3932.803250] Lustre: lustre-MDT0000: new disk, initializing [ 3932.824523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3932.831934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3934.118148] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3938.023453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3938.062489] Lustre: 156102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3938.085034] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3938.087754] Lustre: Skipped 1 previous similar message [ 3938.122918] Lustre: lustre-MDT0001: new disk, initializing [ 3938.154223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3938.167319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3938.170670] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3939.442287] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3942.979518] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3944.731690] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3946.934880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3947.046333] Lustre: lustre-OST0000: new disk, initializing [ 3947.048072] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3947.050637] Lustre: 158069:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3947.086366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3948.405442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3948.408189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3948.443331] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3949.000522] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3952.584098] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3953.907762] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3953.968151] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3955.322062] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3955.400826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3958.758255] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3958.759177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3958.760122] LustreError: Skipped 1 previous similar message [ 3958.764793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3958.767741] Lustre: Skipped 1 previous similar message [ 3962.122431] Lustre: server umount lustre-OST0000 complete [ 3963.874151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3963.874345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3963.875069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3963.882351] Lustre: Skipped 2 previous similar messages [ 3968.993111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3968.995158] Lustre: Skipped 2 previous similar messages [ 3969.443154] Lustre: server umount lustre-MDT0000 complete [ 3970.638076] LustreError: 156894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787102460 with bad export cookie 14410134653486836917 [ 3970.640404] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3970.641794] LustreError: 156894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3970.748526] Lustre: server umount lustre-MDT0001 complete [ 3971.876942] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 21:21:00 (1787102460) [ 3972.366539] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3972.932769] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 21:21:01 (1787102461) [ 3975.340631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3975.503590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3976.754582] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3979.534536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3981.053893] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3983.505147] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3985.397064] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3987.758174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3989.937831] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 3992.544419] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4005.345962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4005.347382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4005.349062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4005.359155] Lustre: Skipped 1 previous similar message [ 4011.110487] Lustre: server umount lustre-MDT0000 complete [ 4012.440320] LustreError: 160456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787102502 with bad export cookie 14410134653486837785 [ 4012.444816] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4012.446100] LustreError: 160456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4013.269727] LustreError: 160472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4013.274760] LustreError: 160472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4015.585146] LustreError: 160473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4015.585500] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4015.595384] Lustre: Skipped 1 previous similar message [ 4015.597799] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4015.599401] Lustre: Skipped 7 previous similar messages [ 4017.635077] LustreError: 162966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4018.626241] Lustre: server umount lustre-MDT0001 complete [ 4020.826598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4024.907163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4024.992084] LustreError: 164333:0:(llog.c:1655:llog_backup()) MGC192.168.204.130@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4024.994640] Lustre: 164333:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.130@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4042.208917] LustreError: 164338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4042.214045] LustreError: 164338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4047.329068] LustreError: 164338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4047.335514] LustreError: 164338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4057.569082] LustreError: 164339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4057.573051] LustreError: 164339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 4074.710109] LustreError: 164339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4074.716677] LustreError: 164339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 4085.856175] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4085.857740] Lustre: 164333:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.130@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4108.772933] LustreError: 162305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4108.788687] LustreError: 162305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 21 previous similar messages [ 4126.698494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4126.702093] Lustre: Skipped 2 previous similar messages [ 4128.173636] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4131.209688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4131.813429] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4133.280134] LustreError: 165075:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 4133.284061] LustreError: 165075:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000b951f16e x1873912621265280/t0(0) o250->MGC192.168.204.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1787102623 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4133.291648] LustreError: 165075:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 4133.856875] LustreError: 147554:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000039267402 x1873912621270016/t0(0) o250->MGC192.168.204.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4135.495038] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4138.432322] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4138.985024] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4139.023566] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4140.943273] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4141.553949] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4147.256982] Lustre: server umount lustre-MDT0000 complete [ 4148.551451] LustreError: 160456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787102638 with bad export cookie 14410134653486844225 [ 4148.555389] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4148.557207] LustreError: 160456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4148.682362] Lustre: server umount lustre-MDT0001 complete [ 4169.516671] Lustre: server umount lustre-OST0000 complete [ 4177.861623] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 4181.646445] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 4195.217309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4195.306303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4195.318493] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4195.357440] Lustre: lustre-MDT0000: new disk, initializing [ 4195.379129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4195.381484] Lustre: Skipped 1 previous similar message [ 4195.386908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4196.931882] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4201.785334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4201.829123] Lustre: 169405:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4201.949429] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4203.625267] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4208.172316] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4210.470364] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4213.393122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4213.524061] Lustre: 171377:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4216.266591] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4220.435604] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4220.482814] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4220.979985] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4222.864949] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4222.963148] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4224.807858] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4224.908054] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4230.628358] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4230.628464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4230.632107] LustreError: Skipped 1 previous similar message [ 4230.640733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4230.643700] Lustre: Skipped 3 previous similar messages [ 4240.352150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4240.419061] Lustre: server umount lustre-OST0000 complete [ 4242.912986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4256.224209] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4256.287353] Lustre: server umount lustre-MDT0000 complete [ 4257.865220] LustreError: 169442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787102747 with bad export cookie 14410134653486861025 [ 4257.868570] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4257.869880] LustreError: 169442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4258.014908] Lustre: server umount lustre-MDT0001 complete [ 4261.592855] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 21:25:50 (1787102750) [ 4264.054514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4264.220295] LustreError: 173787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4264.228175] LustreError: 173787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [ 4265.794498] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4268.695648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4270.331726] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4273.291426] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4275.412368] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4278.058488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4280.417057] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4283.236067] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4297.853101] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 21:26:26 (1787102786) [ 4299.745261] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4299.747848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4299.752935] Lustre: Skipped 3 previous similar messages [ 4299.754887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4299.757263] Lustre: Skipped 9 previous similar messages [ 4304.986749] Lustre: server umount lustre-OST0000 complete [ 4312.402395] Lustre: server umount lustre-MDT0000 complete [ 4313.673495] LustreError: 173767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787102803 with bad export cookie 14410134653486861893 [ 4313.675727] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4313.678461] LustreError: 173767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4313.790541] Lustre: server umount lustre-MDT0001 complete [ 4317.491874] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 4318.591459] Key type lgssc unregistered [ 4318.727313] LNet: 177533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4318.729615] LNetError: 177533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4318.736271] LNet: Removed LNI 192.168.204.130@tcp [ 4319.062957] Key type .llcrypt unregistered [ 4319.064174] Key type ._llcrypt unregistered [ 4326.541555] Key type ._llcrypt registered [ 4326.543058] Key type .llcrypt registered [ 4326.585949] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 4326.937887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4326.966472] alg: No test for adler32 (adler32-zlib) [ 4327.821566] Lustre: Lustre: Build Version: 2.17.57_1_g147bb23 [ 4327.912905] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 4329.496177] Key type lgssc registered [ 4329.899030] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4332.400044] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4332.406156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4347.808389] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4352.928292] LustreError: 178753:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.130@tcp: failed processing log, type 4: rc = -110 [ 4358.048415] LustreError: 178753:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4358.051666] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4358.053257] LustreError: MGC192.168.204.130@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4358.056013] LustreError: 178753:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4358.058352] LustreError: 178753:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 4358.060363] LustreError: 178753:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4358.076913] Lustre: server umount lustre-OST0000 complete [ 4358.078015] LustreError: 178753:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4360.936295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4362.079212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4363.432068] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4366.240609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4366.361739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4367.690135] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4370.010754] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4371.791781] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4374.053087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4374.133921] LustreError: 181063:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4374.138382] Lustre: 181063:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.130@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4374.190104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4376.187320] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4378.493970] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4381.669391] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4381.669406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4381.671402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4381.674013] Lustre: Skipped 2 previous similar messages [ 4381.679515] Lustre: Skipped 2 previous similar messages [ 4385.302659] Lustre: server umount lustre-MDT0000 complete [ 4386.538102] LustreError: 179275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787102876 with bad export cookie 15099107277234791835 [ 4386.540964] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4386.542023] LustreError: 179275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4386.655520] Lustre: server umount lustre-MDT0001 complete [ 4398.106717] Lustre: server umount lustre-OST0000 complete [ 4401.070652] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 4404.431094] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 4417.028184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4417.112257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4417.122704] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4417.157163] Lustre: lustre-MDT0000: new disk, initializing [ 4417.176734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4417.183154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4418.586182] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4422.846487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4422.888281] Lustre: 184579:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4422.922432] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4422.924921] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4422.927076] Lustre: Skipped 1 previous similar message [ 4422.965130] Lustre: lustre-MDT0001: new disk, initializing [ 4422.998164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4423.012022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4423.015298] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4424.273606] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4428.005850] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4429.988895] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4432.409228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4432.508332] Lustre: lustre-OST0000: new disk, initializing [ 4432.509977] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4432.512504] Lustre: 186547:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4432.541138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4434.593611] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4436.404950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4436.407605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4436.439085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4438.279794] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4439.641359] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4439.727846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4441.072691] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4441.148154] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4446.690989] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4446.691576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4446.695024] LustreError: Skipped 1 previous similar message [ 4446.700181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4446.703589] Lustre: Skipped 1 previous similar message [ 4447.888023] Lustre: server umount lustre-OST0000 complete [ 4451.809936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4451.811173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4451.814637] Lustre: Skipped 2 previous similar messages [ 4455.255637] Lustre: server umount lustre-MDT0000 complete [ 4456.499078] LustreError: 184572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787102946 with bad export cookie 15099107277234792535 [ 4456.501867] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4456.502225] LustreError: 184572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4456.611076] Lustre: server umount lustre-MDT0001 complete [ 4459.333977] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 21:29:07 (1787102947) [ 4464.964830] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 4466.070191] Key type lgssc unregistered [ 4466.207404] LNet: 189492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4466.210515] LNetError: 189492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4466.218630] LNet: Removed LNI 192.168.204.130@tcp [ 4466.541158] Key type .llcrypt unregistered [ 4466.542211] Key type ._llcrypt unregistered [ 4473.404934] Key type ._llcrypt registered [ 4473.406621] Key type .llcrypt registered [ 4473.443371] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 4473.838499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4473.877366] alg: No test for adler32 (adler32-zlib) [ 4474.728729] Lustre: Lustre: Build Version: 2.17.57_1_g147bb23 [ 4474.811875] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 4476.392164] Key type lgssc registered [ 4476.727657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4479.143078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4479.150173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4480.296798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4481.625225] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4484.420754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4484.535521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4485.812920] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4488.057513] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4489.722806] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4492.212808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4492.303601] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4492.304696] LustreError: 192547:0:(llog.c:1696:llog_backup()) MGC192.168.204.130@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4492.306978] Lustre: 192547:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.130@tcp: can't backup local config lustre-OST0000: rc = -28 [ 4492.343282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4494.172699] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4496.411672] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4497.071393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4497.072405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4497.075290] Lustre: Skipped 2 previous similar messages [ 4497.077361] Lustre: Skipped 1 previous similar message [ 4502.497702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4502.499703] Lustre: Skipped 3 previous similar messages [ 4503.189247] Lustre: server umount lustre-MDT0000 complete [ 4504.375127] LustreError: 190666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787102994 with bad export cookie 2021861829657531937 [ 4504.377647] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4504.378281] LustreError: 190666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4504.480228] Lustre: server umount lustre-MDT0001 complete [ 4515.932290] Lustre: server umount lustre-OST0000 complete [ 4518.875997] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 4521.891703] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 4533.455645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4533.552387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4533.564816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4533.603274] Lustre: lustre-MDT0000: new disk, initializing [ 4533.629446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4533.635959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4535.008489] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4539.133581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4539.177744] Lustre: 196076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4539.201228] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4539.202964] Lustre: Skipped 1 previous similar message [ 4539.204153] LustreError: 196857:0:(llog.c:1696:llog_backup()) MGC192.168.204.130@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 4539.207317] LustreError: 196857:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4539.209589] Lustre: 196857:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.130@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 4539.218531] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4539.220763] Lustre: Skipped 1 previous similar message [ 4539.260809] Lustre: lustre-MDT0001: new disk, initializing [ 4539.304269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4539.317168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4539.320468] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4540.682354] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4544.415161] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4546.220716] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4548.487949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4548.599681] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4548.600861] Lustre: Skipped 1 previous similar message [ 4548.602044] LustreError: 198020:0:(llog.c:1696:llog_backup()) MGC192.168.204.130@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4548.604848] LustreError: 198020:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4548.607306] Lustre: 198020:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.130@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 4548.611213] Lustre: 198020:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 4548.619888] Lustre: lustre-OST0000: new disk, initializing [ 4548.621332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4548.623436] Lustre: 198045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4548.655674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4550.568202] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4554.205232] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4555.530596] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4556.791037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4556.794317] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4556.823976] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4557.624509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4558.999822] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4559.072533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4561.892476] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4561.892662] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4561.894603] LustreError: Skipped 1 previous similar message [ 4561.898754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4561.900958] Lustre: Skipped 1 previous similar message [ 4565.840238] Lustre: server umount lustre-OST0000 complete [ 4570.080945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4570.083822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4570.087978] Lustre: Skipped 1 previous similar message [ 4570.089969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4572.130168] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4577.250042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4577.251769] Lustre: Skipped 3 previous similar messages [ 4581.344152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4581.417847] Lustre: server umount lustre-MDT0000 complete [ 4582.683112] LustreError: 196068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787103072 with bad export cookie 2021861829657532637 [ 4582.685826] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4582.686523] LustreError: 196068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4582.802591] Lustre: server umount lustre-MDT0001 complete [ 4585.753718] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 21:31:14 (1787103074) [ 4590.929062] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 4594.247321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4594.379104] LustreError: 201183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4594.386271] LustreError: 201183:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4594.391828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4595.697538] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4598.320150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4599.746702] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4600.646780] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4602.757832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4604.765762] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4607.548623] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4607.609881] Lustre: lustre-OST0001: new disk, initializing [ 4607.611588] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4607.613784] Lustre: 203492:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4609.039971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4609.044140] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4609.081167] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4609.557918] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4614.868875] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4619.577137] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4620.196085] Lustre: DEBUG MARKER: ADD OST3 [ 4621.801769] loop: module loaded [ 4621.819307] loop0: detected capacity change from 0 to 204800000 [ 4621.891263] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4621.894752] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4621.900328] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4623.904738] loop0: detected capacity change from 0 to 204800000 [ 4625.797551] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4625.860384] Lustre: lustre-OST0003: new disk, initializing [ 4625.862194] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 4625.865110] Lustre: 205219:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4626.842257] Lustre: DEBUG MARKER: STOP OST3 [ 4627.228399] Lustre: *** cfs_fail_loc=2109, val=2*** [ 4627.254956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 4627.258991] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 4627.270931] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 4627.508333] Lustre: Failing over lustre-OST0003 [ 4627.568186] Lustre: server umount lustre-OST0003 complete [ 4627.990359] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4629.280125] LustreError: 205529:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 4631.900733] Lustre: DEBUG MARKER: START OST3 again [ 4632.548694] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4632.549887] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4632.551946] LustreError: 202670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4632.555349] Lustre: Skipped 1 previous similar message [ 4632.564109] LustreError: 202670:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4634.028233] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4634.100527] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 4634.103294] Lustre: Skipped 4 previous similar messages [ 4634.106776] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 4635.362845] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4635.873225] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4635.933733] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4635.937314] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4649.086627] Lustre: Failing over lustre-OST0003 [ 4649.130716] Lustre: server umount lustre-OST0003 complete [ 4649.449444] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4649.451853] LustreError: 206795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4649.453382] Lustre: Skipped 1 previous similar message [ 4649.457462] LustreError: 206795:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4654.565143] LustreError: 206795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4654.571696] LustreError: 206795:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4659.686430] LustreError: 202671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4659.690749] LustreError: 202671:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4664.806758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4664.809339] Lustre: Skipped 5 previous similar messages [ 4667.680816] Lustre: server umount lustre-MDT0000 complete [ 4668.931440] LustreError: 201165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787103158 with bad export cookie 2021861829657533505 [ 4668.934943] LustreError: 201165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4668.935331] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4669.921180] LustreError: 201917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4669.921938] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4669.927222] LustreError: 201917:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4669.930695] Lustre: Skipped 5 previous similar messages [ 4675.250487] Lustre: server umount lustre-MDT0001 complete [ 4676.607153] Lustre: server umount lustre-OST0000 complete [ 4677.954712] Lustre: server umount lustre-OST0001 complete [ 4680.183282] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 4683.034132] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 4696.314831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4696.416817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4696.427935] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4696.468711] Lustre: lustre-MDT0000: new disk, initializing [ 4696.497514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4697.905889] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4702.056648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4702.100222] Lustre: 210632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4702.199968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4702.202346] Lustre: Skipped 1 previous similar message [ 4702.214603] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4703.599111] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4707.354428] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4709.326230] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4711.606263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4711.734900] Lustre: 212598:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4713.145491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4713.148209] Lustre: Skipped 1 previous similar message [ 4713.149866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4713.182872] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4713.814595] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4717.691519] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4719.206716] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4719.296259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4720.774170] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4720.848585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4723.685718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4723.686759] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4723.693295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4723.694721] LustreError: Skipped 1 previous similar message [ 4723.696467] Lustre: Skipped 4 previous similar messages [ 4727.643320] Lustre: server umount lustre-OST0000 complete [ 4732.896796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4735.132596] Lustre: server umount lustre-MDT0000 complete [ 4736.314248] LustreError: 211436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787103226 with bad export cookie 2021861829657535759 [ 4736.317257] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4736.318826] LustreError: 211436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4736.428407] Lustre: server umount lustre-MDT0001 complete [ 4737.542870] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 21:33:46 (1787103226) [ 4742.351939] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 4745.366786] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 4757.140857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4757.234752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4757.246302] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4757.248298] Lustre: Skipped 3 previous similar messages [ 4757.289359] Lustre: lustre-MDT0000: new disk, initializing [ 4757.290803] Lustre: Skipped 2 previous similar messages [ 4757.318302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4758.788387] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4762.973863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4763.017826] Lustre: 217200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4763.131464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4764.625721] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4768.326236] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4770.028099] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4772.195329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4772.302085] Lustre: 219166:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4773.797114] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4774.328309] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4778.030674] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4779.417099] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4779.503451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4780.835874] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4780.905719] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4784.101927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4784.103877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4784.106195] Lustre: Skipped 5 previous similar messages [ 4787.603345] Lustre: server umount lustre-OST0000 complete [ 4789.218782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4789.222639] Lustre: Skipped 6 previous similar messages [ 4794.971286] Lustre: server umount lustre-MDT0000 complete [ 4796.134165] LustreError: 217191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787103285 with bad export cookie 2021861829657536641 [ 4796.137433] LustreError: 217191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4796.138027] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4796.253724] Lustre: server umount lustre-MDT0001 complete [ 4798.901674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4799.044435] LustreError: 221379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4799.052793] LustreError: 221379:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4800.391014] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4803.026201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4804.405309] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4806.692477] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4808.411492] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4810.557683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4812.558426] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4814.877309] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4833.249033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4833.251436] LustreError: Skipped 1 previous similar message [ 4837.779538] Lustre: server umount lustre-OST0000 complete [ 4845.147173] Lustre: server umount lustre-MDT0000 complete [ 4845.537154] LustreError: 222106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4845.544592] LustreError: 222106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4846.449499] LustreError: 221360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787103336 with bad export cookie 2021861829657537523 [ 4846.453064] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4846.453477] LustreError: 221360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4846.573219] Lustre: server umount lustre-MDT0001 complete [ 4949.918027] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 4950.996921] Key type lgssc unregistered [ 4951.143435] LNet: 225063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4951.146385] LNetError: 225063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4951.156449] LNet: Removed LNI 192.168.204.130@tcp [ 4951.502119] Key type .llcrypt unregistered [ 4951.503702] Key type ._llcrypt unregistered [ 4960.011700] Key type ._llcrypt registered [ 4960.013125] Key type .llcrypt registered [ 4960.056785] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 4966.461498] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 4966.950425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4966.983336] alg: No test for adler32 (adler32-zlib) [ 4967.852230] Lustre: Lustre: Build Version: 2.17.57_1_g147bb23 [ 4967.950268] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 4969.528191] Key type lgssc registered [ 4969.952141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4981.459393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4981.464672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4982.551164] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4982.564261] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4982.598918] Lustre: lustre-MDT0000: new disk, initializing [ 4982.618788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4982.624745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4983.937982] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4987.937361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4987.978440] Lustre: 228384:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4988.001620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4988.003405] Lustre: Skipped 1 previous similar message [ 4988.037469] Lustre: lustre-MDT0001: new disk, initializing [ 4988.067260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4988.080129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4988.083940] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4989.434181] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 4993.073880] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4994.808988] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4997.029046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4997.131418] Lustre: lustre-OST0000: new disk, initializing [ 4997.132967] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4997.135136] Lustre: 230353:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4997.160073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4999.008230] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5002.618370] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5003.255550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5003.258915] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5003.294773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5003.983929] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5004.067455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5005.401629] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5005.481312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5008.353552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5008.354977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5008.356451] LustreError: Skipped 1 previous similar message [ 5008.362613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5008.365518] Lustre: Skipped 1 previous similar message [ 5012.177951] Lustre: server umount lustre-OST0000 complete [ 5013.473565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5013.474182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5013.477064] Lustre: Skipped 2 previous similar messages [ 5018.593144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5018.596659] Lustre: Skipped 2 previous similar messages [ 5019.492053] Lustre: server umount lustre-MDT0000 complete [ 5020.797733] LustreError: 230352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787103510 with bad export cookie 17271709638492675474 [ 5020.802425] LustreError: 230352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5020.803738] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5020.943110] Lustre: server umount lustre-MDT0001 complete [ 5026.799665] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 5029.967821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5030.113522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5031.395434] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5034.098906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5034.230136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5035.674870] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5036.654342] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5038.853532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5040.987266] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5043.876605] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5043.955704] Lustre: lustre-OST0001: new disk, initializing [ 5043.957266] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5043.959628] Lustre: 235587:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5043.996699] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5043.999986] Lustre: Skipped 1 previous similar message [ 5046.027261] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5051.398565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5051.404594] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5051.447491] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5051.579863] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5058.420768] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5062.251296] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 21:39:10 (1787103550) [ 5065.185317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5065.187697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5065.192143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5069.088493] Lustre: server umount lustre-MDT0000 complete [ 5070.348512] LustreError: 235586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787103560 with bad export cookie 17271709638492676342 [ 5070.354540] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5070.357673] LustreError: 235586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5070.469872] Lustre: server umount lustre-MDT0001 complete [ 5082.053408] Lustre: server umount lustre-OST0000 complete [ 5093.607537] Lustre: server umount lustre-OST0001 complete [ 5095.779196] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 5099.311107] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 5111.606150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5111.702672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5111.716273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5111.756631] Lustre: lustre-MDT0000: new disk, initializing [ 5111.782435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5111.789180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5113.232790] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5117.590650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5117.631191] Lustre: 239527:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5117.746198] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5119.020836] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5122.588720] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5124.360993] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5126.515796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5126.634122] Lustre: lustre-OST0000: new disk, initializing [ 5126.635792] Lustre: Skipped 1 previous similar message [ 5126.637430] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5126.639742] Lustre: Skipped 2 previous similar messages [ 5126.642086] Lustre: 241496:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5127.936107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5127.939027] Lustre: Skipped 1 previous similar message [ 5127.940951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5127.977091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5128.598083] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5132.175517] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5133.710206] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5133.791319] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5135.166294] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5135.241786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5138.402263] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5138.403437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5138.404802] LustreError: Skipped 1 previous similar message [ 5138.408624] Lustre: Skipped 3 previous similar messages [ 5138.410324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5138.412743] Lustre: Skipped 5 previous similar messages [ 5148.645354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5148.647794] Lustre: Skipped 3 previous similar messages [ 5150.176145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5150.236990] Lustre: server umount lustre-OST0000 complete [ 5153.761584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5153.761717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5157.598818] Lustre: server umount lustre-MDT0000 complete [ 5158.815213] LustreError: 241495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787103648 with bad export cookie 17271709638492678463 [ 5158.817735] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5158.820748] LustreError: 241495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5158.934782] Lustre: server umount lustre-MDT0001 complete [ 5161.674923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5161.831102] LustreError: 243711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5161.836393] LustreError: 243711:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5161.856476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5161.859560] Lustre: Skipped 2 previous similar messages [ 5163.177618] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5165.944571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5167.370260] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5169.674919] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5171.380359] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5173.466141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5175.470586] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5177.781390] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5320.160779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5320.162822] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5320.165688] Lustre: Skipped 1 previous similar message [ 5320.167216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5320.168717] Lustre: Skipped 2 previous similar messages [ 5325.774775] Lustre: server umount lustre-OST0000 complete [ 5333.079815] Lustre: server umount lustre-MDT0000 complete [ 5333.473286] LustreError: 246213:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5333.477603] LustreError: 246213:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5334.238422] LustreError: 243737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787103824 with bad export cookie 17271709638492679345 [ 5334.241440] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5334.242958] LustreError: 243737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5334.369714] Lustre: server umount lustre-MDT0001 complete [ 5442.417342] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 5443.940392] Key type lgssc unregistered [ 5444.104565] LNet: 247322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5444.109869] LNetError: 247322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5444.117977] LNet: Removed LNI 192.168.204.130@tcp [ 5444.580858] Key type .llcrypt unregistered [ 5444.582457] Key type ._llcrypt unregistered [ 5457.560817] Key type ._llcrypt registered [ 5457.563634] Key type .llcrypt registered [ 5457.630438] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 5465.748465] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 5466.308824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5466.320699] alg: No test for adler32 (adler32-zlib) [ 5467.266657] Lustre: Lustre: Build Version: 2.17.57_1_g147bb23 [ 5467.409247] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 5469.016539] Key type lgssc registered [ 5469.573878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5491.122235] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 5495.886620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5495.897720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5497.036133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5497.057144] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5497.112210] Lustre: lustre-MDT0000: new disk, initializing [ 5497.154590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5497.164887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5499.503796] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5504.927473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5504.987814] Lustre: 251366:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5505.019238] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5505.022258] Lustre: Skipped 1 previous similar message [ 5505.068221] Lustre: lustre-MDT0001: new disk, initializing [ 5505.120308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5505.145052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5505.150947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5507.175284] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5509.882426] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5513.388122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5513.545762] Lustre: lustre-OST0000: new disk, initializing [ 5513.548846] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5513.553368] Lustre: 252999:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5513.606658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5516.564982] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5517.260289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5517.266133] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5517.335196] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5522.214222] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5522.319729] Lustre: lustre-OST0001: new disk, initializing [ 5522.322899] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5522.328491] Lustre: 253872:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5522.393191] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5524.314966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5524.319821] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5524.366740] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5525.298404] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5533.151232] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5544.970381] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5555.172118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5555.173543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5555.180281] Lustre: Skipped 3 previous similar messages [ 5555.183323] Lustre: Skipped 2 previous similar messages [ 5558.035616] Lustre: server umount lustre-MDT0000 complete [ 5560.407824] LustreError: 252998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787104050 with bad export cookie 15216332111076619201 [ 5560.412351] LustreError: 252998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5560.412857] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5560.628574] Lustre: server umount lustre-MDT0001 complete [ 5572.644045] Lustre: server umount lustre-OST0000 complete [ 5584.397540] Lustre: server umount lustre-OST0001 complete [ 5593.081198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5599.075788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5603.820664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5609.661373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5621.684239] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 5628.491104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5628.504154] Lustre: lustre-MDT0000: reset Object Index mappings [ 5628.825705] LustreError: 258876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5628.842129] LustreError: 258876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5628.881163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5631.607702] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5634.019382] LustreError: 258877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5636.792268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5636.804944] Lustre: lustre-MDT0001: reset Object Index mappings [ 5637.077845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5639.652650] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5641.635168] Lustre: 260018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5645.847558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5647.079351] LustreError: 260476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5649.730117] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5652.456955] LustreError: 260373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5653.491224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5654.428772] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5654.596854] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5654.601834] Lustre: Skipped 1 previous similar message [ 5657.931436] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5659.635982] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 5662.445859] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5664.331598] Lustre: 261887:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5667.813616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5667.818286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5667.830579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5667.834306] Lustre: Skipped 1 previous similar message [ 5669.858450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5669.860112] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5669.867710] Lustre: Skipped 2 previous similar messages [ 5672.956629] Lustre: server umount lustre-MDT0000 complete [ 5674.790464] LustreError: 258857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787104164 with bad export cookie 15216332111076625998 [ 5674.793135] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5674.798195] LustreError: 258857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5674.955757] Lustre: server umount lustre-MDT0001 complete [ 5687.076468] Lustre: server umount lustre-OST0000 complete [ 5698.359404] Lustre: server umount lustre-OST0001 complete [ 5707.656746] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 21:49:56 (1787104196) [ 5715.940615] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 5720.993479] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 5738.910881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5739.027706] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5739.041514] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5739.080288] Lustre: lustre-MDT0000: new disk, initializing [ 5739.124243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5739.134224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5741.409294] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5747.520499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5747.585381] Lustre: 266141:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5747.625114] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5747.631312] Lustre: Skipped 1 previous similar message [ 5747.678519] Lustre: lustre-MDT0001: new disk, initializing [ 5747.754845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5747.764719] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5749.823115] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5755.025797] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5757.851662] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5761.490884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5761.646045] Lustre: 268111:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5763.403086] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5763.489919] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5764.808401] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5770.002756] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5772.301885] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5772.429850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5774.841623] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5774.981746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5778.914714] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5778.915256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5778.922760] LustreError: Skipped 1 previous similar message [ 5778.932770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5778.938989] Lustre: Skipped 4 previous similar messages [ 5782.182385] Lustre: server umount lustre-OST0000 complete [ 5788.642581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5788.647212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5788.654380] Lustre: Skipped 1 previous similar message [ 5788.658158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5798.370618] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5798.374984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5798.384309] Lustre: Skipped 5 previous similar messages [ 5798.468286] Lustre: server umount lustre-MDT0000 complete [ 5800.545608] LustreError: 266134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787104290 with bad export cookie 15216332111076628084 [ 5800.550105] LustreError: 266134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5800.551581] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5800.749672] Lustre: server umount lustre-MDT0001 complete [ 5809.994907] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 5815.009601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5815.261919] LustreError: 271040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5815.277071] LustreError: 271040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5815.288059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5815.292145] Lustre: Skipped 2 previous similar messages [ 5817.476208] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5822.108898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5824.370857] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5825.934610] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5829.198443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5831.879735] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5835.564831] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5835.653461] Lustre: lustre-OST0001: new disk, initializing [ 5835.655625] Lustre: Skipped 1 previous similar message [ 5835.658368] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5835.661402] Lustre: Skipped 1 previous similar message [ 5835.664923] Lustre: 273347:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5837.408751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5837.411966] Lustre: Skipped 1 previous similar message [ 5837.414420] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5837.458894] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5838.361303] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5843.486756] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5844.912804] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5847.119344] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 5847.133518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 5851.109574] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5851.112571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5851.118150] Lustre: Skipped 3 previous similar messages [ 5851.125367] Lustre: Skipped 3 previous similar messages [ 5855.785681] Lustre: server umount lustre-MDT0000 complete [ 5856.226630] LustreError: 271036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5856.233035] LustreError: 271036:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5857.130540] LustreError: 271021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787104346 with bad export cookie 15216332111076628966 [ 5857.133445] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5857.135709] LustreError: 271021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5857.260405] Lustre: server umount lustre-MDT0001 complete [ 5868.548880] Lustre: server umount lustre-OST0000 complete [ 5880.094486] Lustre: server umount lustre-OST0001 complete [ 5885.411180] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 5888.808222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5888.943206] LustreError: 276399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5890.329109] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5893.390052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5895.059124] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5896.045349] Lustre: 277540:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5898.322766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5899.501749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 5900.478841] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5903.629585] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5905.802761] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5908.980465] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 5909.160694] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5910.410409] Lustre: 279413:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5912.272779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 5912.275088] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 5915.635869] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 21:53:24 (1787104404) [ 5929.445237] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5929.447154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5929.447735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5929.450075] Lustre: Skipped 4 previous similar messages [ 5929.455636] Lustre: Skipped 3 previous similar messages [ 5933.286407] Lustre: server umount lustre-MDT0000 complete [ 5934.562757] LustreError: 277131:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5934.572755] LustreError: 277131:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 5934.722776] LustreError: 276382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787104424 with bad export cookie 15216332111076635280 [ 5934.726563] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5934.727145] LustreError: 276382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5940.913709] Lustre: server umount lustre-MDT0001 complete [ 5948.416154] Lustre: server umount lustre-OST0000 complete [ 5955.740103] Lustre: server umount lustre-OST0001 complete [ 5958.054607] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 5961.899723] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 5978.124229] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 5982.114922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5982.209617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5982.220934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5982.260683] Lustre: lustre-MDT0000: new disk, initializing [ 5982.284207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5982.286391] Lustre: Skipped 7 previous similar messages [ 5982.292284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5983.752957] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5988.054385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5988.093387] Lustre: 283644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5988.199052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5989.684445] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 5994.905939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5995.012769] Lustre: 285278:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5995.345291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5997.116877] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6001.648224] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6003.508562] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6003.899504] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6008.605094] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6015.055635] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 21:55:03 (1787104503) [ 6019.043593] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6019.045738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6019.050165] Lustre: Skipped 4 previous similar messages [ 6019.052850] Lustre: Skipped 3 previous similar messages [ 6019.056468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6022.006574] Lustre: server umount lustre-MDT0000 complete [ 6023.570251] LustreError: 285277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787104513 with bad export cookie 15216332111076643162 [ 6023.576050] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6023.577060] LustreError: 285277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6023.707563] Lustre: server umount lustre-MDT0001 complete [ 6035.592168] Lustre: server umount lustre-OST0000 complete [ 6046.307888] Lustre: server umount lustre-OST0001 complete [ 6049.598018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6049.786274] LustreError: 288390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6049.795810] LustreError: 288390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6051.400795] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6054.696989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6056.518329] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6059.296198] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6061.235756] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6063.658375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6066.174434] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6069.050511] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6080.956562] loop1: detected capacity change from 0 to 204800000 [ 6081.038914] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6081.045837] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6081.050798] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6082.990258] loop1: detected capacity change from 0 to 204800000 [ 6085.100833] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6085.138869] LustreError: 288379:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs8695: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 6085.147675] LustreError: 288379:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs8695 [ 6085.151481] LustreError: 288379:0:(mgs_handler.c:517:mgs_target_reg()) fs8695-MDT0001: Error checking target index: rc = -13 [ 6085.155855] LustreError: fs8695-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6085.159987] LustreError: 292105:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 6085.163869] LustreError: 292105:0:(tgt_mount.c:2129:server_put_super()) no obd fs8695-MDT0001 [ 6085.166563] LustreError: 292105:0:(tgt_mount.c:116:server_deregister_mount()) fs8695-MDT0001 not registered [ 6085.180681] Lustre: server umount fs8695-MDT0001 complete [ 6085.182367] LustreError: 292105:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6088.723190] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6088.768160] Lustre: Setting parameter fs8695-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs8695-MDT0001 [ 6088.775264] Lustre: Skipped 2 previous similar messages [ 6088.798433] Lustre: srv-fs8695-MDT0001: No data found on store. Initialize space. [ 6088.800985] Lustre: Skipped 4 previous similar messages [ 6088.841643] Lustre: fs8695-MDT0001: new disk, initializing [ 6088.843716] Lustre: Skipped 3 previous similar messages [ 6088.887147] Lustre: srv-fs8695-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 6088.890983] Lustre: 292570:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs8695-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6088.895673] LustreError: 292570:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs8695-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6090.435542] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6093.723714] Lustre: server umount fs8695-MDT0001 complete [ 6098.726241] loop1: detected capacity change from 0 to 204800000 [ 6100.913381] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6101.050800] Lustre: 294137:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs8695-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6101.054734] LustreError: 294137:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs8695-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6102.605059] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6104.300194] Lustre: server umount fs8695-MDT0001 complete [ 6110.179029] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6110.184104] Lustre: Skipped 3 previous similar messages [ 6114.078296] Lustre: server umount lustre-MDT0000 complete [ 6117.243317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6117.295882] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6118.930227] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6121.876531] loop1: detected capacity change from 0 to 204800000 [ 6122.471798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6123.899056] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6124.023856] Lustre: 296578:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs8695-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6124.027235] LustreError: 296578:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs8695-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6125.490756] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6127.518268] Lustre: server umount fs8695-MDT0001 complete [ 6136.802229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6141.544964] Lustre: server umount lustre-OST0000 complete [ 6148.066668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6148.069730] Lustre: Skipped 10 previous similar messages [ 6149.242235] Lustre: server umount lustre-MDT0000 complete [ 6150.775553] LustreError: 288372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787104640 with bad export cookie 15216332111076645836 [ 6150.778225] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6150.783374] LustreError: 288372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6150.915670] Lustre: server umount lustre-MDT0001 complete [ 6155.404089] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 6156.505929] Key type lgssc unregistered [ 6156.651497] LNet: 298920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6156.655242] LNetError: 298920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6156.666434] LNet: Removed LNI 192.168.204.130@tcp [ 6157.030143] Key type .llcrypt unregistered [ 6157.032041] Key type ._llcrypt unregistered [ 6165.765549] Key type ._llcrypt registered [ 6165.766779] Key type .llcrypt registered [ 6165.810647] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 6166.258908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6166.269869] alg: No test for adler32 (adler32-zlib) [ 6167.140989] Lustre: Lustre: Build Version: 2.17.57_1_g147bb23 [ 6167.240621] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 6168.824133] Key type lgssc registered [ 6169.210632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6172.027618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6172.035460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6173.223042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6174.877342] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6178.150275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6178.325438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6179.769988] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6182.266642] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6184.229190] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6186.473274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6186.597652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6188.591042] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6188.645478] LustreError: 301996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6188.654211] LustreError: 301996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6191.076067] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6194.147033] LustreError: 301996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6199.269637] LustreError: 302372:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6199.277679] LustreError: 302372:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6201.568530] LustreError: 301996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6206.679533] LustreError: 302671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6206.687023] LustreError: 302671:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6208.190867] loop1: detected capacity change from 0 to 204800000 [ 6208.253609] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6208.258712] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6208.264659] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6210.231060] loop1: detected capacity change from 0 to 204800000 [ 6212.227764] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6212.288147] Lustre: fs14096-OST0000: new disk, initializing [ 6212.290832] Lustre: srv-fs14096-OST0000: No data found on store. Initialize space. [ 6212.294519] Lustre: 303924:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6212.321287] Lustre: fs14096-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6214.481731] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6216.920082] LustreError: 301999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6216.928193] LustreError: 301999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6218.393603] Lustre: server umount fs14096-OST0000 complete [ 6222.993290] loop1: detected capacity change from 0 to 204800000 [ 6225.100318] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6225.196922] Lustre: fs14096-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6227.466743] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6230.875301] loop2: detected capacity change from 0 to 204800000 [ 6230.957806] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6230.962500] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6230.967577] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6232.833784] loop2: detected capacity change from 0 to 204800000 [ 6234.921290] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6234.952685] LustreError: fs14096-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 6234.957957] LustreError: fs14096-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6234.962345] LustreError: 306960:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 6234.965049] LustreError: 306960:0:(tgt_mount.c:2129:server_put_super()) no obd fs14096-OST0001 [ 6234.967774] LustreError: 306960:0:(tgt_mount.c:116:server_deregister_mount()) fs14096-OST0001 not registered [ 6234.979455] Lustre: server umount fs14096-OST0001 complete [ 6234.981193] LustreError: 306960:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6235.110990] LustreError: 301999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6235.117238] LustreError: 301999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6238.813557] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6238.869894] Lustre: fs14096-OST0001: new disk, initializing [ 6238.872990] Lustre: srv-fs14096-OST0001: No data found on store. Initialize space. [ 6238.877327] Lustre: 307510:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6238.907028] Lustre: fs14096-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6241.008157] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6244.693620] Lustre: server umount fs14096-OST0000 complete [ 6248.329414] Lustre: server umount fs14096-OST0001 complete [ 6252.648040] loop1: detected capacity change from 0 to 204800000 [ 6254.772852] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6257.017723] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6259.468141] loop2: detected capacity change from 0 to 204800000 [ 6261.526059] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6261.599605] Lustre: fs14096-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6261.602196] Lustre: Skipped 1 previous similar message [ 6263.638222] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6266.984434] Lustre: server umount fs14096-OST0000 complete [ 6268.118863] LustreError: 306984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6268.126067] LustreError: 306984:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 6270.460610] Lustre: server umount fs14096-OST0001 complete [ 6276.066514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6276.066899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6276.070219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6276.070304] LustreError: Skipped 1 previous similar message [ 6276.078392] Lustre: Skipped 1 previous similar message [ 6280.597986] Lustre: server umount lustre-OST0000 complete [ 6285.793871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6285.799186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6285.806105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6285.808975] Lustre: Skipped 1 previous similar message [ 6288.028719] Lustre: server umount lustre-MDT0000 complete [ 6289.462441] LustreError: 300913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787104779 with bad export cookie 13305932562342097432 [ 6289.466275] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6289.468445] LustreError: 300913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6289.601859] Lustre: server umount lustre-MDT0001 complete [ 6293.644327] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 6294.720943] Key type lgssc unregistered [ 6294.860757] LNet: 313421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6294.864393] LNetError: 313421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6294.872502] LNet: Removed LNI 192.168.204.130@tcp [ 6295.206148] Key type .llcrypt unregistered [ 6295.207766] Key type ._llcrypt unregistered [ 6306.600707] Key type ._llcrypt registered [ 6306.602183] Key type .llcrypt registered [ 6306.644787] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 6313.591178] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 6314.029740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6314.082681] alg: No test for adler32 (adler32-zlib) [ 6314.946034] Lustre: Lustre: Build Version: 2.17.57_1_g147bb23 [ 6315.041855] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 6316.624115] Key type lgssc registered [ 6317.043569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6330.649961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6330.658652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6331.748211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6331.760821] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6331.799393] Lustre: lustre-MDT0000: new disk, initializing [ 6331.826959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6331.833795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6333.315538] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6338.040555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6338.094297] Lustre: 317252:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6338.124836] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6338.128083] Lustre: Skipped 1 previous similar message [ 6338.173884] Lustre: lustre-MDT0001: new disk, initializing [ 6338.220612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6338.239709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6338.244200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6340.004542] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6344.252633] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6346.358347] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6349.136134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6349.249835] Lustre: lustre-OST0000: new disk, initializing [ 6349.251945] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6349.255532] Lustre: 319220:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6349.294199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6351.427929] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6351.436150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6351.440675] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6351.490782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6355.562128] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6356.350836] Lustre: Mounted lustre-client - version 2.17.57_1_g147bb23 [ 6356.709987] Lustre: Unmounted lustre-client [ 6361.571126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6361.572069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6361.576506] Lustre: Skipped 1 previous similar message [ 6361.580772] Lustre: Skipped 1 previous similar message [ 6362.463568] Lustre: server umount lustre-OST0000 complete [ 6364.129821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6364.134385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6364.142751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6366.690527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6366.690948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6370.142306] Lustre: server umount lustre-MDT0000 complete [ 6371.738553] LustreError: 317243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787104861 with bad export cookie 16225847095944049716 [ 6371.742315] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6371.743727] LustreError: 317243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6371.908385] Lustre: server umount lustre-MDT0001 complete [ 6376.005961] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 6377.083639] Key type lgssc unregistered [ 6407.968374] LNet: 321392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6407.971353] LNetError: 321392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6485.798386] LNet: Removed LNI 192.168.204.130@tcp [ 6486.157981] Key type .llcrypt unregistered [ 6486.159528] Key type ._llcrypt unregistered [ 6492.911395] Key type ._llcrypt registered [ 6492.912712] Key type .llcrypt registered [ 6492.955607] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 6500.327979] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 6500.824474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6500.834423] alg: No test for adler32 (adler32-zlib) [ 6501.706161] Lustre: Lustre: Build Version: 2.17.57_1_g147bb23 [ 6501.808552] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 6503.400171] Key type lgssc registered [ 6503.854661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6516.452303] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 22:03:24 (1787105004) [ 6519.443134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6519.453583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6520.579408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6520.596986] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6520.646873] Lustre: lustre-MDT0000: new disk, initializing [ 6520.681951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6520.689141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6522.403976] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6527.572405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6527.624403] Lustre: 324593:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6527.653952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6527.657182] Lustre: Skipped 1 previous similar message [ 6527.697567] Lustre: lustre-MDT0001: new disk, initializing [ 6527.745039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6527.765942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6527.773479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6529.474766] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6534.297509] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6536.843361] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6539.473684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6539.575687] Lustre: lustre-OST0000: new disk, initializing [ 6539.577705] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6539.580702] Lustre: 326556:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6539.609162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6541.660103] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6545.483620] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6545.913990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6545.917402] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6545.948631] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6556.130586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6556.130929] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6556.131697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6556.132954] LustreError: Skipped 1 previous similar message [ 6556.136328] Lustre: Skipped 1 previous similar message [ 6561.250693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6561.252571] Lustre: Skipped 2 previous similar messages [ 6562.197193] Lustre: server umount lustre-OST0000 complete [ 6563.808876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6563.810969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6563.815014] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6566.369162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6566.369599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6569.563537] Lustre: server umount lustre-MDT0000 complete [ 6570.782102] LustreError: 324583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787105060 with bad export cookie 17976816309286507300 [ 6570.785334] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6570.785673] LustreError: 324583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6570.898674] Lustre: server umount lustre-MDT0001 complete [ 6574.218291] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 6575.166225] Key type lgssc unregistered [ 6575.285325] LNet: 328466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6575.287943] LNetError: 328466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6575.299254] LNet: Removed LNI 192.168.204.130@tcp [ 6575.575136] Key type .llcrypt unregistered [ 6575.576361] Key type ._llcrypt unregistered [ 6583.447285] Key type ._llcrypt registered [ 6583.448677] Key type .llcrypt registered [ 6583.482806] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 6583.778408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6583.806184] alg: No test for adler32 (adler32-zlib) [ 6584.652975] Lustre: Lustre: Build Version: 2.17.57_1_g147bb23 [ 6584.737805] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 6586.313096] Key type lgssc registered [ 6586.658841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6589.208641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6589.214310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6590.363578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6591.653954] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6594.357696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6594.476390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6595.777064] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6598.188386] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6600.024230] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6602.292836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6602.431880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6604.257090] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6606.655968] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6620.522186] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6620.996671] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6622.427156] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.30@tcp (stopping) [ 6624.468359] Lustre: server umount lustre-MDT0000 complete [ 6625.248982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6625.251539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6625.256423] LustreError: 330458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6625.261633] LustreError: 330458:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6625.687227] LustreError: 329743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787105115 with bad export cookie 7357479566843464828 [ 6625.690846] LustreError: 329743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6625.691032] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6626.273185] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6626.273702] LustreError: 329713:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6626.277648] Lustre: Skipped 1 previous similar message [ 6626.283970] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6627.542258] LustreError: 329741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6627.548790] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.30@tcp (stopping) [ 6630.304756] LustreError: 329711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6630.305058] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6631.849270] Lustre: server umount lustre-MDT0001 complete [ 6637.789275] Lustre: lustre-OST0000: Not available for connect from 192.168.204.30@tcp (stopping) [ 6646.997709] Lustre: lustre-OST0000: Not available for connect from 192.168.204.30@tcp (stopping) [ 6647.001245] Lustre: Skipped 1 previous similar message [ 6647.264118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6647.317664] Lustre: server umount lustre-OST0000 complete [ 6650.097477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6651.334202] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6652.190485] Lustre: 333914:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.130@tcp [ 6652.854295] Lustre: server umount MGS complete [ 6655.687496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6655.841152] LustreError: 334485:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6655.864122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6657.073227] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6659.787076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6659.903815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6661.307050] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6664.093725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6666.171502] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6668.600368] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6674.168752] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 6675.921746] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6678.422157] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6679.525721] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6679.525874] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6679.530142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6679.534610] Lustre: Skipped 1 previous similar message [ 6685.587105] Lustre: server umount lustre-OST0000 complete [ 6689.761762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6689.766152] Lustre: Skipped 2 previous similar messages [ 6693.079880] Lustre: server umount lustre-MDT0000 complete [ 6694.342326] LustreError: 334467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787105184 with bad export cookie 7357479566843465654 [ 6694.345351] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6694.346158] LustreError: 334467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6694.457840] Lustre: server umount lustre-MDT0001 complete [ 6698.001980] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 6698.958611] Key type lgssc unregistered [ 6699.080338] LNet: 337922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6699.083246] LNetError: 337922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6699.092278] LNet: Removed LNI 192.168.204.130@tcp [ 6699.377911] Key type .llcrypt unregistered [ 6699.379604] Key type ._llcrypt unregistered [ 6704.980099] Key type ._llcrypt registered [ 6704.981152] Key type .llcrypt registered [ 6705.020340] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 6711.147877] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 6711.481785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6711.535716] alg: No test for adler32 (adler32-zlib) [ 6712.383054] Lustre: Lustre: Build Version: 2.17.57_1_g147bb23 [ 6712.469679] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 6714.056162] Key type lgssc registered [ 6714.376857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6725.287312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6725.293456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6726.372813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6726.382331] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6726.417324] Lustre: lustre-MDT0000: new disk, initializing [ 6726.439276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6726.446327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6727.694547] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6731.735994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6731.776165] Lustre: 341052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6731.797933] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6731.799815] Lustre: Skipped 1 previous similar message [ 6731.831610] Lustre: lustre-MDT0001: new disk, initializing [ 6731.860322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6731.872282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6731.875775] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6733.145374] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6736.696502] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6738.475767] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6740.527042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6740.623337] Lustre: lustre-OST0000: new disk, initializing [ 6740.625412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6740.629309] Lustre: 343022:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6740.656819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6742.415992] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6746.208988] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6746.616609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6746.621232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6746.655853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6747.719784] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6747.793861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6749.238971] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6749.320900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6751.715861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6751.715927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6751.716644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6751.721744] Lustre: Skipped 1 previous similar message [ 6752.081805] Lustre: server umount lustre-OST0000 complete [ 6756.835065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6756.835938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6756.839643] Lustre: Skipped 1 previous similar message [ 6756.841280] Lustre: Skipped 1 previous similar message [ 6759.381824] Lustre: server umount lustre-MDT0000 complete [ 6760.586854] LustreError: 341047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787105250 with bad export cookie 6078865927519266250 [ 6760.589717] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6760.592385] LustreError: 341047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6760.697852] Lustre: server umount lustre-MDT0001 complete [ 6763.522855] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 22:07:32 (1787105252) [ 6768.604261] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 6772.025570] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 6783.502192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6783.577129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6783.587595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6783.623602] Lustre: lustre-MDT0000: new disk, initializing [ 6783.642224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6783.647816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6784.896231] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6788.755232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6788.790954] Lustre: 347770:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6788.809909] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6788.812062] Lustre: Skipped 1 previous similar message [ 6788.843580] Lustre: lustre-MDT0001: new disk, initializing [ 6788.870445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6788.880753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6788.883757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6790.301854] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6794.110124] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6796.075961] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6798.307361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6798.395214] Lustre: lustre-OST0000: new disk, initializing [ 6798.396775] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6798.398750] Lustre: 349739:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6798.425351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6800.327256] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 6803.972904] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6806.519274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6806.522345] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6806.552104] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6821.857480] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6821.859354] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6821.861023] LustreError: Skipped 2 previous similar messages [ 6821.865374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6821.868551] Lustre: Skipped 2 previous similar messages [ 6824.403752] Lustre: server umount lustre-OST0000 complete [ 6825.644096] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6825.646197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6825.649499] Lustre: Skipped 1 previous similar message [ 6825.651711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6825.741476] Lustre: server umount lustre-MDT0000 complete [ 6827.002181] LustreError: 347761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787105316 with bad export cookie 6078865927519267118 [ 6827.006143] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6827.006798] LustreError: 347761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6827.124346] Lustre: server umount lustre-MDT0001 complete [ 6830.663600] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 6831.626035] Key type lgssc unregistered [ 6831.754280] LNet: 351888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6831.756645] LNetError: 351888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6831.766265] LNet: Removed LNI 192.168.204.130@tcp [ 6832.082800] Key type .llcrypt unregistered [ 6832.084160] Key type ._llcrypt unregistered [ 6863.311231] Key type ._llcrypt registered [ 6863.312281] Key type .llcrypt registered [ 6863.347204] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 6863.449323] Key type .llcrypt unregistered [ 6863.450653] Key type ._llcrypt unregistered [ 6891.073949] Key type ._llcrypt registered [ 6891.075289] Key type .llcrypt registered [ 6891.113415] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 6891.243280] Key type .llcrypt unregistered [ 6891.244510] Key type ._llcrypt unregistered [ 6902.118545] Key type ._llcrypt registered [ 6902.119818] Key type .llcrypt registered [ 6902.154156] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 6902.256299] Key type .llcrypt unregistered [ 6902.257515] Key type ._llcrypt unregistered [ 6912.986701] Key type ._llcrypt registered [ 6912.988066] Key type .llcrypt registered [ 6913.023332] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 6913.142183] Key type .llcrypt unregistered [ 6913.143164] Key type ._llcrypt unregistered [ 6939.869576] Key type ._llcrypt registered [ 6939.870644] Key type .llcrypt registered [ 6939.907607] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 6940.023194] Key type .llcrypt unregistered [ 6940.024212] Key type ._llcrypt unregistered [ 6950.727511] Key type ._llcrypt registered [ 6950.728844] Key type .llcrypt registered [ 6950.769597] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 6950.878417] Key type .llcrypt unregistered [ 6950.879660] Key type ._llcrypt unregistered [ 6961.425899] Key type ._llcrypt registered [ 6961.427649] Key type .llcrypt registered [ 6961.461642] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 6961.565351] Key type .llcrypt unregistered [ 6961.566760] Key type ._llcrypt unregistered [ 6972.604356] Key type ._llcrypt registered [ 6972.605398] Key type .llcrypt registered [ 6972.640518] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 6972.754427] Key type .llcrypt unregistered [ 6972.756357] Key type ._llcrypt unregistered [ 6983.606849] Key type ._llcrypt registered [ 6983.607982] Key type .llcrypt registered [ 6983.645408] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 6983.743183] Key type .llcrypt unregistered [ 6983.744327] Key type ._llcrypt unregistered [ 6994.844548] Key type ._llcrypt registered [ 6994.845517] Key type .llcrypt registered [ 6994.878028] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 6994.975270] Key type .llcrypt unregistered [ 6994.976635] Key type ._llcrypt unregistered [ 7015.262355] Key type ._llcrypt registered [ 7015.263570] Key type .llcrypt registered [ 7015.298623] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 7015.408451] Key type .llcrypt unregistered [ 7015.410157] Key type ._llcrypt unregistered [ 7026.415243] Key type ._llcrypt registered [ 7026.416401] Key type .llcrypt registered [ 7026.458563] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 7026.579432] Key type .llcrypt unregistered [ 7026.580504] Key type ._llcrypt unregistered [ 7038.089939] Key type ._llcrypt registered [ 7038.091846] Key type .llcrypt registered [ 7038.134206] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 7038.253555] Key type .llcrypt unregistered [ 7038.255242] Key type ._llcrypt unregistered [ 7057.253558] Key type ._llcrypt registered [ 7057.254742] Key type .llcrypt registered [ 7057.292691] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 7063.846691] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 7064.186635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7064.221573] alg: No test for adler32 (adler32-zlib) [ 7065.068535] Lustre: Lustre: Build Version: 2.17.57_1_g147bb23 [ 7065.150720] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 7066.728155] Key type lgssc registered [ 7067.099352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7078.107215] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7078.113682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7079.210466] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 7079.223882] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 7079.264692] Lustre: tst250fs-MDT0000: new disk, initializing [ 7079.308400] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7079.315344] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7080.665469] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 7084.657092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7084.699080] Lustre: 371841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7084.724107] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 7084.727162] Lustre: Skipped 1 previous similar message [ 7084.765945] Lustre: tst250fs-MDT0001: new disk, initializing [ 7084.798412] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7084.809766] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7084.813478] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7086.149113] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 7088.373145] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7090.521380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7090.608689] Lustre: tst250fs-OST0000: new disk, initializing [ 7090.610491] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 7090.613125] Lustre: 373475:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7090.641836] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7092.515527] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 7094.203081] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7094.207123] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7094.246245] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7096.442633] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7096.510492] Lustre: tst250fs-OST0001: new disk, initializing [ 7096.512745] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 7096.515262] Lustre: 374347:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7096.550620] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7098.380312] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug -1 all [ 7100.101096] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7100.105454] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7100.131691] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7100.902384] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 7115.748811] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7115.749899] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7115.751966] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 7115.754684] Lustre: Skipped 2 previous similar messages [ 7117.223276] Lustre: server umount tst250fs-MDT0000 complete [ 7118.536505] LustreError: 371880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787105608 with bad export cookie 5937341315513076385 [ 7118.538422] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7118.541427] LustreError: 371880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7118.663780] Lustre: server umount tst250fs-MDT0001 complete [ 7130.049647] Lustre: server umount tst250fs-OST0000 complete [ 7140.378216] Lustre: server umount tst250fs-OST0001 complete [ 7146.192448] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 7148.863781] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 7158.431297] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 22:14:07 (1787105647) [ 7158.913635] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 7163.962247] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 7166.903678] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 7177.472244] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6918 sec ========== 22:14:26 (1787105666) [ 7178.130561] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:14:26 (1787105666) === [ 7179.382829] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:14:27 (1787105667) === [ 7193.280377] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 7194.316170] Key type lgssc unregistered [ 7194.440325] LNet: 383458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7194.444239] LNetError: 383458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7194.454272] LNet: Removed LNI 192.168.204.130@tcp [ 7194.794103] Key type .llcrypt unregistered [ 7194.795290] Key type ._llcrypt unregistered