[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 473672883 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003162] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008432] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010008] pid_max: default: 32768 minimum: 301 [ 0.012011] LSM: Security Framework initializing [ 0.014008] Yama: becoming mindful. [ 0.015040] SELinux: Initializing. [ 0.016061] *** VALIDATE selinux *** [ 0.024867] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029796] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032091] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033127] *** VALIDATE tmpfs *** [ 0.035351] *** VALIDATE proc *** [ 0.037053] *** VALIDATE cgroup *** [ 0.038010] *** VALIDATE cgroup2 *** [ 0.039280] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040161] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043021] Spectre V2 : User space: Vulnerable [ 0.044011] Speculative Store Bypass: Vulnerable [ 0.046787] debug: unmapping init [mem 0xffffffffb3259000-0xffffffffb3260fff] [ 0.048178] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049719] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050024] ... version: 2 [ 0.051014] ... bit width: 48 [ 0.052016] ... generic registers: 4 [ 0.053017] ... value mask: 0000ffffffffffff [ 0.054016] ... max period: 00007fffffffffff [ 0.055018] ... fixed-purpose events: 3 [ 0.056017] ... event mask: 000000070000000f [ 0.057285] rcu: Hierarchical SRCU implementation. [ 0.059400] smp: Bringing up secondary CPUs ... [ 0.060561] x86: Booting SMP configuration: [ 0.061027] .... node #0, CPUs: #1 #2 #3 [ 0.064394] smp: Brought up 1 node, 4 CPUs [ 0.066015] smpboot: Max logical packages: 1 [ 0.067023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.241069] node 0 deferred pages initialised in 171ms [ 0.246014] devtmpfs: initialized [ 0.247259] x86/mm: Memory block size: 128MB [ 0.249776] gcov: version magic: 0x41383552 [ 0.250570] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.255083] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.259308] pinctrl core: initialized pinctrl subsystem [ 0.262178] [ 0.262914] ************************************************************* [ 0.267033] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.271013] ** ** [ 0.274013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.278015] ** ** [ 0.282015] ** This means that this kernel is built to expose internal ** [ 0.285013] ** IOMMU data structures, which may compromise security on ** [ 0.288013] ** your system. ** [ 0.292015] ** ** [ 0.296016] ** If you see this message and you are not debugging the ** [ 0.299013] ** kernel, report this immediately to your vendor! ** [ 0.303015] ** ** [ 0.306016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.310056] ************************************************************* [ 0.316557] NET: Registered protocol family 16 [ 0.318526] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.321088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.325091] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.329047] cpuidle: using governor menu [ 0.330846] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.333489] PCI: Using configuration type 1 for base access [ 0.336128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.346061] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.348048] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.354052] cryptd: max_cpu_qlen set to 1000 [ 0.357271] ACPI: Added _OSI(Module Device) [ 0.359025] ACPI: Added _OSI(Processor Device) [ 0.361014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.363015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.368529] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.374305] ACPI: Interpreter enabled [ 0.376063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.378013] ACPI: Using IOAPIC for interrupt routing [ 0.379094] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.383536] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.398833] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.400059] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.403029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.407106] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.412493] acpiphp: Slot [2] registered [ 0.415243] acpiphp: Slot [5] registered [ 0.416271] acpiphp: Slot [6] registered [ 0.418201] acpiphp: Slot [7] registered [ 0.420208] acpiphp: Slot [8] registered [ 0.422298] acpiphp: Slot [9] registered [ 0.424210] acpiphp: Slot [10] registered [ 0.426198] acpiphp: Slot [3] registered [ 0.428153] acpiphp: Slot [4] registered [ 0.429128] acpiphp: Slot [11] registered [ 0.432129] acpiphp: Slot [12] registered [ 0.434143] acpiphp: Slot [13] registered [ 0.436203] acpiphp: Slot [14] registered [ 0.439122] acpiphp: Slot [15] registered [ 0.441122] acpiphp: Slot [16] registered [ 0.443142] acpiphp: Slot [17] registered [ 0.446124] acpiphp: Slot [18] registered [ 0.448111] acpiphp: Slot [19] registered [ 0.450129] acpiphp: Slot [20] registered [ 0.452106] acpiphp: Slot [21] registered [ 0.453101] acpiphp: Slot [22] registered [ 0.455123] acpiphp: Slot [23] registered [ 0.457111] acpiphp: Slot [24] registered [ 0.458108] acpiphp: Slot [25] registered [ 0.460153] acpiphp: Slot [26] registered [ 0.462142] acpiphp: Slot [27] registered [ 0.464166] acpiphp: Slot [28] registered [ 0.465123] acpiphp: Slot [29] registered [ 0.468185] acpiphp: Slot [30] registered [ 0.470195] acpiphp: Slot [31] registered [ 0.471061] PCI host bridge to bus 0000:00 [ 0.473038] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.476041] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.478035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.481043] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.486040] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.489038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.491224] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.495068] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.498636] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.512014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.518119] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.522021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.526018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.530026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.532690] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.536951] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.540243] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.543901] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.550025] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.563021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.570023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.578276] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.589017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.598016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.619024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.633354] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.641020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.650020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.670023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.682713] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.692021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.700025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.719018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.731546] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.739020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.751025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.768019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.779355] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.786025] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.792024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.812022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.823283] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.829022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.836021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.853022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.864909] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.867396] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.869344] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.871338] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.874234] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.880048] iommu: Default domain type: Passthrough [ 0.882546] SCSI subsystem initialized [ 0.884211] ACPI: bus type USB registered [ 0.886183] usbcore: registered new interface driver usbfs [ 0.889117] usbcore: registered new interface driver hub [ 0.890078] usbcore: registered new device driver usb [ 0.892174] pps_core: LinuxPPS API ver. 1 registered [ 0.893009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.896055] PTP clock support registered [ 0.898133] EDAC MC: Ver: 3.0.0 [ 0.900193] PCI: Using ACPI for IRQ routing [ 0.901637] NetLabel: Initializing [ 0.903019] NetLabel: domain hash size = 128 [ 0.904010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.906104] NetLabel: unlabeled traffic allowed by default [ 0.909094] vgaarb: loaded [ 0.910290] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.912012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.917370] clocksource: Switched to clocksource kvm-clock [ 1.024824] VFS: Disk quotas dquot_6.6.0 [ 1.026611] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.029201] *** VALIDATE ramfs *** [ 1.030456] *** VALIDATE hugetlbfs *** [ 1.032172] pnp: PnP ACPI init [ 1.034610] pnp: PnP ACPI: found 6 devices [ 1.057647] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.061348] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.063749] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.066323] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.068871] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.072874] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.076184] NET: Registered protocol family 2 [ 1.078685] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.083638] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.087609] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.094501] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.098137] TCP: Hash tables configured (established 65536 bind 65536) [ 1.101645] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.105240] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.108356] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.110951] NET: Registered protocol family 1 [ 1.114086] RPC: Registered named UNIX socket transport module. [ 1.116860] RPC: Registered udp transport module. [ 1.118491] RPC: Registered tcp transport module. [ 1.120023] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.121855] NET: Registered protocol family 44 [ 1.123216] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.124938] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.126569] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.128567] PCI: CLS 0 bytes, default 64 [ 1.130209] Unpacking initramfs... [ 2.578441] debug: unmapping init [mem 0xffff92167cc54000-0xffff92167ffbffff] [ 2.583075] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.585743] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.589285] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.081279] Initialise system trusted keyrings [ 3.083149] Key type blacklist registered [ 3.085208] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.094373] zbud: loaded [ 3.097544] *** VALIDATE nfs *** [ 3.099035] *** VALIDATE nfs4 *** [ 3.100832] pstore: using deflate compression [ 3.104750] Platform Keyring initialized [ 3.226849] NET: Registered protocol family 38 [ 3.228976] Key type asymmetric registered [ 3.230746] Asymmetric key parser 'x509' registered [ 3.233033] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.236676] io scheduler mq-deadline registered [ 3.239700] io scheduler kyber registered [ 3.241572] io scheduler bfq registered [ 3.243588] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.246811] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.250310] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.253745] ACPI: Power Button [PWRF] [ 3.260722] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.268462] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.282529] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.292388] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.320594] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.350354] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.378360] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.384845] Non-volatile memory driver v1.3 [ 3.386908] Linux agpgart interface v0.103 [ 3.427527] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.432518] vda: detected capacity change from 0 to 74698752 [ 3.450285] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.452469] vdb: detected capacity change from 0 to 1073741824 [ 3.467129] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.470213] vdc: detected capacity change from 0 to 2621440000 [ 3.483488] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.486881] vdd: detected capacity change from 0 to 2621440000 [ 3.502653] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.506539] vde: detected capacity change from 0 to 4294967296 [ 3.521601] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.524513] vdf: detected capacity change from 0 to 4294967296 [ 3.530561] libphy: Fixed MDIO Bus: probed [ 3.534847] usbcore: registered new interface driver usbserial_generic [ 3.537282] usbserial: USB Serial support registered for generic [ 3.539753] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.544535] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.546161] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.550047] mousedev: PS/2 mouse device common for all mice [ 3.552630] rtc_cmos 00:05: RTC can wake from S4 [ 3.555589] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.556036] rtc_cmos 00:05: registered as rtc0 [ 3.560702] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.561599] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.565703] intel_pstate: CPU model not supported [ 3.571626] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.571963] hid: raw HID events driver (C) Jiri Kosina [ 3.576541] usbcore: registered new interface driver usbhid [ 3.578304] usbhid: USB HID core driver [ 3.579465] drop_monitor: Initializing network drop monitor service [ 3.581831] Initializing XFRM netlink socket [ 3.583683] NET: Registered protocol family 10 [ 3.586655] Segment Routing with IPv6 [ 3.588898] NET: Registered protocol family 17 [ 3.591442] mpls_gso: MPLS GSO support [ 3.596671] RAS: Correctable Errors collector initialized. [ 3.599319] AVX version of gcm_enc/dec engaged. [ 3.601315] AES CTR mode by8 optimization enabled [ 3.689176] sched_clock: Marking stable (3689153435, 0)->(4585853567, -896700132) [ 3.692816] registered taskstats version 1 [ 3.695202] Loading compiled-in X.509 certificates [ 3.696962] zswap: loaded using pool lzo/zbud [ 3.723403] Key type big_key registered [ 3.736673] Key type encrypted registered [ 3.738520] ima: No TPM chip found, activating TPM-bypass! [ 3.740875] ima: Allocated hash algorithm: sha1 [ 3.742740] ima: No architecture policies found [ 3.744675] evm: Initialising EVM extended attributes: [ 3.746781] evm: security.selinux [ 3.748291] evm: security.ima [ 3.749538] evm: security.capability [ 3.751092] evm: HMAC attrs: 0x1 [ 3.753938] rtc_cmos 00:05: setting system clock to 2026-09-03 13:05:38 UTC (1788440738) [ 3.761238] debug: unmapping init [mem 0xffffffffb4203000-0xffffffffb43fffff] [ 3.764745] debug: unmapping init [mem 0xffffffffb2f82000-0xffffffffb3258fff] [ 3.774205] Write protecting the kernel read-only data: 28672k [ 3.777712] debug: unmapping init [mem 0xffffffffb1603000-0xffffffffb17fffff] [ 3.781611] debug: unmapping init [mem 0xffffffffb1f14000-0xffffffffb1ffffff] [ 3.818859] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.828950] systemd[1]: Detected virtualization kvm. [ 3.830769] systemd[1]: Detected architecture x86-64. [ 3.833104] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.857865] systemd[1]: No hostname configured. [ 3.859661] systemd[1]: Set hostname to . [ 3.862099] random: systemd: uninitialized urandom read (16 bytes read) [ 3.864708] systemd[1]: Initializing machine ID from random generator. [ 3.921895] random: ln: uninitialized urandom read (6 bytes read) [ 4.006994] random: systemd: uninitialized urandom read (16 bytes read) [ 4.010690] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.015862] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.021631] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.693973] device-mapper: uevent: version 1.0.3 [ 4.696608] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [[ 5.065239] random: fast init done  OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.455965] virtio_net virtio0 ens2: renamed from eth0 [ 5.502270] scsi host0: ata_piix [ 5.519461] scsi host1: ata_piix [ 5.521262] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.525308] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.311183] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.256632] random: crng init done [ 10.258664] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.685418] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.810959] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.053391] SELinux: Disabled at runtime. [ 12.112304] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.123834] systemd[1]: Detected virtualization kvm. [ 12.125823] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.631574] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.634921] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.639421] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.644280] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.649433] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.657663] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.664719] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ 12.825886] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.068489] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.419762] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.438328] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.531378] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.554057] EDAC sbridge: Ver: 1.1.2 [ 15.078706] Key type dns_resolver registered [ 15.380860] NFS: Registering the id_resolver key type [ 15.384343] Key type id_resolver registered [ 15.387130] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg347-server login: [ 41.448954] libcfs: loading out-of-tree module taints kernel. [ 41.469908] Key type ._llcrypt registered [ 41.471359] Key type .llcrypt registered [ 41.523556] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 55.410630] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 57.445936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 57.454917] alg: No test for adler32 (adler32-zlib) [ 58.942656] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 59.886615] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 61.680161] Key type lgssc registered [ 62.691661] hrtimer: interrupt took 3652782 ns [ 63.611102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 80.987980] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 124.933254] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 137.097645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 137.124126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 138.563316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 138.603176] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 138.694466] Lustre: lustre-MDT0000: new disk, initializing [ 138.788700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 138.812721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 143.697170] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 156.642106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 156.733293] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 156.755163] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 156.758747] Lustre: Skipped 1 previous similar message [ 156.832400] Lustre: lustre-MDT0001: new disk, initializing [ 156.903896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 156.926353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 156.935916] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 161.110151] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 165.782873] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 174.258332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 174.569740] Lustre: lustre-OST0000: new disk, initializing [ 174.581580] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 174.593199] Lustre: 8448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 174.693517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.774964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 177.793980] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 177.905175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 181.929991] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.638423] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 196.781580] Lustre: lustre-OST0001: new disk, initializing [ 196.786837] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 196.795790] Lustre: 9521:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 196.884648] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 202.233776] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.817247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 205.822285] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 205.861958] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 215.270946] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 224.724425] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 232.503820] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing check_logdir /tmp/testlogs/ [ 237.439837] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing yml_node [ 243.285556] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 245.822231] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 248.544927] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 250.183345] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 09:09:43 EDT 2026 [ 267.837456] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 269.076768] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 285.156247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 285.167335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 285.182271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 287.720432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 287.720604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 291.049360] Lustre: server umount lustre-MDT0000 complete [ 294.622190] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441029 with bad export cookie 3205798388659333996 [ 294.631436] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 294.642279] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 297.955031] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 297.957199] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.961391] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 297.961401] Lustre: Skipped 3 previous similar messages [ 297.982625] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 298.030082] Lustre: Skipped 3 previous similar messages [ 300.001221] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 300.001224] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 300.001230] Lustre: Skipped 1 previous similar message [ 300.038956] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 301.095192] Lustre: server umount lustre-MDT0001 complete [ 305.466992] Lustre: server umount lustre-OST0000 complete [ 309.516967] Lustre: server umount lustre-OST0001 complete [ 316.234144] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 324.674772] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 359.521001] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 359.798921] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 359.843817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 359.923801] Lustre: lustre-MDT0000: new disk, initializing [ 359.992236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 360.010491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 363.983266] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 372.725418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 372.799877] Lustre: 15072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 372.846327] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 372.853597] Lustre: Skipped 1 previous similar message [ 372.900709] Lustre: lustre-MDT0001: new disk, initializing [ 372.989600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 373.034801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 373.045816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 376.744655] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 386.542224] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 393.744315] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 402.969662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 403.314456] Lustre: lustre-OST0000: new disk, initializing [ 403.319732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 403.328602] Lustre: 17042:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 403.454278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 405.061909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 405.075349] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 405.291557] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 412.002826] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 421.572119] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 426.481921] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 426.723358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 431.131906] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 431.364628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 436.198073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 436.202299] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 436.208364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 436.224986] Lustre: Skipped 2 previous similar messages [ 439.574058] Lustre: server umount lustre-OST0000 complete [ 444.897637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 444.909614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 444.916992] Lustre: Skipped 1 previous similar message [ 444.920241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 449.466958] Lustre: server umount lustre-MDT0000 complete [ 451.556127] LustreError: 15085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 451.614271] LustreError: 15085:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 453.706497] LustreError: 17041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441188 with bad export cookie 3205798388659335529 [ 453.719901] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 453.731918] LustreError: 17041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 454.380608] Lustre: server umount lustre-MDT0001 complete [ 463.404670] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 09:13:15 (1788441195) [ 484.993321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.485496] LustreError: 19919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.589398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 490.484407] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 490.983733] LustreError: 19920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.011810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 501.227333] LustreError: 19920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.275376] LustreError: 19920:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 508.119476] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 517.782078] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 523.683259] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 525.887851] Lustre: Failing over lustre-MDT0000 [ 526.112596] Lustre: server umount lustre-MDT0000 complete [ 526.955149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 526.961556] LustreError: 19920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.988435] Lustre: Skipped 2 previous similar messages [ 542.700123] Lustre: 3643:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441261/real 1788441261] req@ffff9216f6aff800 x1875316094169472/t0(0) o400->MGC192.168.203.147@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788441277 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 542.743639] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 545.226420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 554.183877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 554.189794] Lustre: Skipped 1 previous similar message [ 558.519877] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 559.593731] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 559.602484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 559.627493] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 567.877501] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 572.642584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 572.647551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 572.661592] Lustre: Skipped 1 previous similar message [ 572.673226] Lustre: Skipped 3 previous similar messages [ 577.946644] Lustre: server umount lustre-MDT0000 complete [ 578.022631] LustreError: 19919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.040346] LustreError: 19919:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 581.812705] LustreError: 19900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441316 with bad export cookie 3205798388659336943 [ 581.816228] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 581.825187] LustreError: 19900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 582.188583] Lustre: server umount lustre-MDT0001 complete [ 590.858309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 595.838725] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 603.767465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 609.040487] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 617.379583] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 623.843908] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 625.962409] Lustre: Failing over lustre-MDT0000 [ 626.317214] Lustre: server umount lustre-MDT0000 complete [ 626.657885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 626.669633] Lustre: Skipped 1 previous similar message [ 642.858597] Lustre: 3643:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441361/real 1788441361] req@ffff9216ffb17480 x1875316094199296/t0(0) o400->MGC192.168.203.147@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788441377 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 642.899048] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 642.928410] LustreError: 23256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 642.955361] LustreError: 23256:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 645.547987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 652.263493] LustreError: 3640:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9216ffb17b80 x1875316094202240/t0(0) o250->MGC192.168.203.147@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 652.686500] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 657.656788] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 657.900324] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 657.902578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 657.917567] Lustre: Skipped 1 previous similar message [ 657.964430] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 667.819419] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 673.251278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 673.270212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 678.170516] Lustre: server umount lustre-MDT0000 complete [ 682.109877] LustreError: 23236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441416 with bad export cookie 3205798388659337881 [ 682.113155] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 682.114848] LustreError: 23236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 682.341292] Lustre: server umount lustre-MDT0001 complete [ 692.537611] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:17:05 (1788441425) [ 710.574571] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 721.179772] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 761.701870] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 772.816786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 773.104845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 773.141612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 773.228525] Lustre: lustre-MDT0000: new disk, initializing [ 773.331362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 773.337672] Lustre: Skipped 3 previous similar messages [ 773.353941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 778.009642] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 789.068140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 789.241097] Lustre: 29631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 789.657309] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 794.943097] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 805.418486] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 811.492265] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 818.902676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 819.186378] Lustre: 31596:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 820.767984] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 820.797380] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 820.907361] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 820.930636] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 825.961723] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 836.043947] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 841.453807] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 864.798950] Lustre: server umount lustre-OST0000 complete [ 871.395388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 871.416727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 871.445609] Lustre: Skipped 2 previous similar messages [ 871.458757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 871.476922] Lustre: Skipped 3 previous similar messages [ 878.692567] Lustre: server umount lustre-MDT0000 complete [ 881.787763] LustreError: 29621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441616 with bad export cookie 3205798388659338287 [ 881.796363] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 881.799845] LustreError: 29621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 891.523182] Lustre: server umount lustre-MDT0001 complete [ 903.136039] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 907.660348] Key type lgssc unregistered [ 908.096618] LNet: 33580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.118724] LNetError: 33580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.156525] LNet: Removed LNI 192.168.203.147@tcp [ 909.712143] Key type .llcrypt unregistered [ 909.715412] Key type ._llcrypt unregistered [ 939.685230] Key type ._llcrypt registered [ 939.686673] Key type .llcrypt registered [ 939.815540] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 940.807837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 940.829376] alg: No test for adler32 (adler32-zlib) [ 941.769466] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 941.941115] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 943.584533] Key type lgssc registered [ 944.530726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 951.525619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 951.559245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 953.023388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 957.355992] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 962.532950] LustreError: 34995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 962.559642] LustreError: 34995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 964.848703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 965.078786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 968.779591] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 976.548532] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 982.599991] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 991.595424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 992.003816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 998.199527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 999.032629] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1007.766886] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1009.385270] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1023.348818] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:22:36 (1788441756) [ 1025.363316] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1027.631293] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1035.771338] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:22:48 (1788441768) [ 1045.950466] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:22:58 (1788441778) [ 1046.974644] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1055.661829] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1055.668621] Lustre: Skipped 2 previous similar messages [ 1057.921529] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:23:10 (1788441790) [ 1060.112769] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1060.121205] Lustre: Skipped 2 previous similar messages [ 1069.742242] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1069.747776] Lustre: Skipped 4 previous similar messages [ 1086.505528] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:23:39 (1788441819) [ 1091.012883] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1095.580568] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1105.445912] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:23:58 (1788441838) [ 1107.817652] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1107.826433] Lustre: Skipped 6 previous similar messages [ 1125.376653] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:24:17 (1788441857) [ 1141.730267] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1141.764418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.793143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1142.765728] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1142.780661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.799187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1145.825436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1145.837080] Lustre: Skipped 1 previous similar message [ 1147.933075] Lustre: server umount lustre-OST0000 complete [ 1154.536508] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1154.547911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1154.576395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1161.189932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1161.198457] Lustre: Skipped 3 previous similar messages [ 1166.304137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1166.535513] Lustre: server umount lustre-MDT0000 complete [ 1170.406093] LustreError: 34996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1170.417131] LustreError: 34996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1171.054646] LustreError: 34977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441905 with bad export cookie 4838438394793513947 [ 1171.064504] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1171.067578] LustreError: 34977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1171.484698] Lustre: server umount lustre-MDT0001 complete [ 1182.013784] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 1184.202522] Key type lgssc unregistered [ 1184.527880] LNet: 42860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1184.532703] LNetError: 42860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1184.564166] LNet: Removed LNI 192.168.203.147@tcp [ 1185.489931] Key type .llcrypt unregistered [ 1185.495281] Key type ._llcrypt unregistered [ 1203.537800] Key type ._llcrypt registered [ 1203.542534] Key type .llcrypt registered [ 1203.645638] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 1219.722235] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 1221.378163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1221.409083] alg: No test for adler32 (adler32-zlib) [ 1222.496945] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1222.793891] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 1224.488638] Key type lgssc registered [ 1225.483144] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1263.213397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1263.257924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1264.576140] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1264.607394] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1264.741258] Lustre: lustre-MDT0000: new disk, initializing [ 1264.866796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1264.907718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1270.263336] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1281.590032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.746966] Lustre: 46014:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1281.860985] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1281.866258] Lustre: Skipped 1 previous similar message [ 1281.967158] Lustre: lustre-MDT0001: new disk, initializing [ 1282.084638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1282.129376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1282.144679] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1286.644206] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1296.945765] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1303.781569] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1311.007588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1311.324596] Lustre: lustre-OST0000: new disk, initializing [ 1311.330564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1311.346442] Lustre: 47988:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1311.469111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.619868] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1321.042715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1321.067768] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1321.163293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1329.035686] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1334.655406] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1334.884359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1340.096430] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1340.336048] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1347.046877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1347.056090] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1347.067473] LustreError: Skipped 1 previous similar message [ 1347.074140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1347.095063] Lustre: Skipped 1 previous similar message [ 1347.126365] Lustre: Skipped 1 previous similar message [ 1348.781124] Lustre: server umount lustre-OST0000 complete [ 1353.700435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1353.707632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1353.719274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1357.282037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1357.284502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1357.322503] Lustre: Skipped 1 previous similar message [ 1359.395270] Lustre: server umount lustre-MDT0000 complete [ 1362.401241] LustreError: 46026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.425060] LustreError: 46026:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1363.068489] LustreError: 47987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442097 with bad export cookie 3330798285991229648 [ 1363.075854] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1363.083646] LustreError: 47987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1363.577485] Lustre: server umount lustre-MDT0001 complete [ 1373.934866] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:28:27 (1788442107) [ 1380.809207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1381.309567] LustreError: 50447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1381.372443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1386.312709] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1386.464978] LustreError: 50448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1391.589398] LustreError: 50447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1395.235776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1395.508445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1400.426663] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1408.543261] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1414.140357] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1420.612921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1420.908249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1427.370838] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1435.487748] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1440.086593] LustreError: 52975:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1441.089068] Lustre: Modifying parameter general.timeout=1 in log params [ 1442.885419] Lustre: Modifying parameter general.timeout=3 in log params [ 1442.888680] Lustre: Skipped 1 previous similar message [ 1444.936872] Lustre: Modifying parameter general.timeout=5 in log params [ 1444.941122] Lustre: Skipped 1 previous similar message [ 1449.607656] Lustre: Modifying parameter general.timeout=10 in log params [ 1449.610566] Lustre: Skipped 4 previous similar messages [ 1458.641885] Lustre: Modifying parameter general.timeout=19 in log params [ 1458.649471] Lustre: Skipped 8 previous similar messages [ 1474.928657] Lustre: Modifying parameter general.timeout=36 in log params [ 1474.934265] Lustre: Skipped 16 previous similar messages [ 1506.969154] Lustre: Modifying parameter general.timeout=70 in log params [ 1506.974775] Lustre: Skipped 33 previous similar messages [ 1571.666603] Lustre: Disabling parameter general.timeout= in log params [ 1571.670397] Lustre: Skipped 59 previous similar messages [ 1575.079381] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:31:48 (1788442308) [ 1578.140257] Lustre: 59929:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1579.807991] Lustre: 60028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1583.071140] Lustre: 60226:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1583.078235] Lustre: 60226:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1616.092881] Lustre: 61798:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1616.100102] Lustre: 61798:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1616.118480] LustreError: 61798:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1619.837992] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:32:33 (1788442353) [ 1633.766636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1633.776685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1633.788400] Lustre: Skipped 2 previous similar messages [ 1633.815640] Lustre: Skipped 2 previous similar messages [ 1638.633208] Lustre: server umount lustre-MDT0000 complete [ 1638.888920] LustreError: 50448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1638.903256] LustreError: 50448:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1642.436616] LustreError: 52288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442377 with bad export cookie 3330798285991230523 [ 1642.440132] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1642.449109] LustreError: 52288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1642.858069] Lustre: server umount lustre-MDT0001 complete [ 1657.793678] Lustre: server umount lustre-OST0000 complete [ 1684.317172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1684.692329] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1685.069962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1690.712353] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1699.879201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1699.981130] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1700.009749] Lustre: 64330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1705.556785] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1713.101383] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1719.193474] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1726.678197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1726.929411] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1727.269605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1727.280778] Lustre: Skipped 1 previous similar message [ 1734.059104] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1743.148437] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1744.880385] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1744.895417] Lustre: Skipped 23 previous similar messages [ 1744.951914] Lustre: 66896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1756.130130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1756.144308] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1756.172517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1758.903889] Lustre: server umount lustre-OST0000 complete [ 1766.885819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1766.886206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1766.887329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1766.887334] Lustre: Skipped 1 previous similar message [ 1766.918204] Lustre: Skipped 2 previous similar messages [ 1768.888891] Lustre: server umount lustre-MDT0000 complete [ 1772.002502] LustreError: 64342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1772.027702] LustreError: 64342:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1772.767253] LustreError: 65082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442507 with bad export cookie 3330798285991233652 [ 1772.776408] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1772.783755] LustreError: 65082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1773.144423] Lustre: server umount lustre-MDT0001 complete [ 1786.198791] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 1789.589929] Key type lgssc unregistered [ 1789.841708] LNet: 68106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1789.848727] LNetError: 68106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1789.868090] LNet: Removed LNI 192.168.203.147@tcp [ 1790.921883] Key type .llcrypt unregistered [ 1790.928851] Key type ._llcrypt unregistered [ 1820.194626] Key type ._llcrypt registered [ 1820.198288] Key type .llcrypt registered [ 1820.381476] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 1821.490157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1821.685968] alg: No test for adler32 (adler32-zlib) [ 1822.945740] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1823.287597] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 1825.032164] Key type lgssc registered [ 1826.531484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1835.087705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1835.109256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1836.735154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1841.525162] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1847.271535] LustreError: 69522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1847.291131] LustreError: 69522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1849.876550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1850.551218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1855.483180] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1863.091623] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1868.995751] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1877.312638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1878.226622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1884.671803] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1893.543572] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1904.551909] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1916.387442] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1916.389546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1916.394208] Lustre: Skipped 1 previous similar message [ 1916.406926] Lustre: Skipped 2 previous similar messages [ 1918.622880] Lustre: server umount lustre-MDT0000 complete [ 1921.512042] LustreError: 69522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1921.553278] LustreError: 69522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1922.330272] LustreError: 69502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442657 with bad export cookie 18018745405883532911 [ 1922.338577] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1922.359239] LustreError: 69502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1922.753465] Lustre: server umount lustre-MDT0001 complete [ 1937.219076] Lustre: server umount lustre-OST0000 complete [ 1968.832959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1969.214488] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1969.288086] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1969.638872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1975.105673] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1983.841908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.961511] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1983.985924] Lustre: 74395:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1984.515944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1989.376348] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1997.226979] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2002.843726] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2009.233887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2009.441189] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2009.787381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2016.142146] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2023.902457] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2025.956493] Lustre: 76968:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2028.452825] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2036.714213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2036.724117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2036.733574] Lustre: Skipped 1 previous similar message [ 2036.740195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2037.221466] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2040.484563] Lustre: server umount lustre-OST0000 complete [ 2045.922207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2045.934221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2045.953411] Lustre: Skipped 1 previous similar message [ 2045.955568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2045.963067] Lustre: Skipped 2 previous similar messages [ 2050.260760] Lustre: server umount lustre-MDT0000 complete [ 2052.578449] LustreError: 74402:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2052.607996] LustreError: 74402:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2053.807099] LustreError: 74387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442788 with bad export cookie 18018745405883533744 [ 2053.812304] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2053.821049] LustreError: 74387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2054.166965] Lustre: server umount lustre-MDT0001 complete [ 2065.086289] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 2067.999475] Key type lgssc unregistered [ 2068.330756] LNet: 78180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2068.347444] LNetError: 78180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2068.364863] LNet: Removed LNI 192.168.203.147@tcp [ 2069.247117] Key type .llcrypt unregistered [ 2069.250248] Key type ._llcrypt unregistered [ 2096.902738] Key type ._llcrypt registered [ 2096.907222] Key type .llcrypt registered [ 2096.999727] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 2098.014762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2098.095907] alg: No test for adler32 (adler32-zlib) [ 2099.217166] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2099.455907] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 2101.128387] Key type lgssc registered [ 2102.337683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2110.447738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2110.486386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2112.015915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2116.662850] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2122.211191] LustreError: 79592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2122.244914] LustreError: 79592:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2124.940054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2125.478353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2129.713387] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2140.358594] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2148.445499] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2156.294948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2156.481862] LustreError: 81417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2157.275142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2162.546480] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2169.748138] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2176.937397] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2178.566459] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2179.981768] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2182.588019] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2182.603361] Lustre: Skipped 1 previous similar message [ 2186.749443] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2186.754336] Lustre: Skipped 2 previous similar messages [ 2194.806645] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2194.815454] Lustre: Skipped 6 previous similar messages [ 2211.241775] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2211.246608] Lustre: Skipped 16 previous similar messages [ 2243.269597] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2243.277415] Lustre: Skipped 31 previous similar messages [ 2288.776801] Lustre: 87696:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2291.930626] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 09:43:44 (1788443024) [ 2293.873772] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2295.698750] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 09:43:48 (1788443028) [ 2300.906521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2300.913446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2300.931282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2302.946083] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2302.957348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2302.964513] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2303.000496] Lustre: Skipped 1 previous similar message [ 2305.523836] Lustre: server umount lustre-OST0000 complete [ 2313.187215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2313.193539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2313.211062] Lustre: Skipped 1 previous similar message [ 2313.231914] Lustre: Skipped 1 previous similar message [ 2316.108497] Lustre: server umount lustre-MDT0000 complete [ 2318.305828] LustreError: 79593:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2318.326990] LustreError: 79593:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2320.747924] LustreError: 79574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443055 with bad export cookie 7428199344467634692 [ 2320.750453] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2320.760886] LustreError: 79574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2321.502024] Lustre: server umount lustre-MDT0001 complete [ 2333.560946] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 2337.502667] Key type lgssc unregistered [ 2337.856845] LNet: 88950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2337.869300] LNetError: 88950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2337.883156] LNet: Removed LNI 192.168.203.147@tcp [ 2339.021665] Key type .llcrypt unregistered [ 2339.025759] Key type ._llcrypt unregistered [ 2344.053634] Key type ._llcrypt registered [ 2344.056463] Key type .llcrypt registered [ 2344.225111] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2349.387322] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules [ 2350.886900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2350.976823] alg: No test for adler32 (adler32-zlib) [ 2352.037565] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2352.319404] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 2354.088303] Key type lgssc registered [ 2355.713189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2357.107936] LustreError: 90168:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2373.677403] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 2378.323313] LustreError: 90168:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2383.265302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2383.284832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2384.862813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.422649] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2395.109978] LustreError: 91177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.132425] LustreError: 91177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2398.632726] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:45:31 (1788443131) [ 2400.233776] LustreError: 91178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.594799] LustreError: 91209:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2403.615598] LustreError: 91209:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2403.622807] LustreError: 91209:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2404.116910] Lustre: server umount lustre-MDT0000 complete [ 2418.217698] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 2421.065417] Key type lgssc unregistered [ 2421.338858] LNet: 92705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2421.346655] LNetError: 92705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2421.360639] LNet: Removed LNI 192.168.203.147@tcp [ 2422.243414] Key type .llcrypt unregistered [ 2422.246078] Key type ._llcrypt unregistered [ 2445.093282] Key type ._llcrypt registered [ 2445.101394] Key type .llcrypt registered [ 2445.255179] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 2446.766086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2446.787908] alg: No test for adler32 (adler32-zlib) [ 2447.835641] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2448.126406] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 2449.784148] Key type lgssc registered [ 2451.365585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2459.763625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2459.772597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2461.146598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2466.271769] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2470.372782] LustreError: 93922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2470.387122] LustreError: 93922:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2475.491701] LustreError: 93923:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2475.495053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2476.312789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2480.971299] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2489.178831] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2494.950075] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2501.467594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2502.098832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2508.180874] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2515.876445] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2568.099753] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:48:21 (1788443301) [ 2569.992913] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2571.956018] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 09:48:25 (1788443305) [ 2576.866836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2576.869814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2576.880483] Lustre: Skipped 2 previous similar messages [ 2576.886178] Lustre: Skipped 1 previous similar message [ 2581.987918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2581.991551] Lustre: Skipped 1 previous similar message [ 2582.289662] Lustre: server umount lustre-MDT0000 complete [ 2585.865169] LustreError: 93904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443320 with bad export cookie 6089462769865006219 [ 2585.876395] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2585.879687] LustreError: 93904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2586.167537] Lustre: server umount lustre-MDT0001 complete [ 2600.449683] Lustre: server umount lustre-OST0000 complete [ 2612.929271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2613.311952] LustreError: 98221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2613.523087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2617.845232] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2618.852453] LustreError: 98222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2623.970761] LustreError: 98221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2628.034537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2628.863991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2633.289278] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2641.174821] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2646.052451] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2659.126972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2659.329951] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2659.339098] LustreError: 98208:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2659.385983] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2659.394973] LustreError: 100303:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2659.400968] LustreError: 100303:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2659.409835] LustreError: 100303:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2659.497812] Lustre: server umount lustre-OST0000 complete [ 2659.500584] LustreError: 100303:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2667.824328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2668.009682] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2668.021588] LustreError: 98208:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2668.032936] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2668.041977] LustreError: 100765:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2668.047345] LustreError: 100765:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2668.053258] LustreError: 100765:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2668.137748] Lustre: server umount lustre-OST0000 complete [ 2668.142994] LustreError: 100765:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2678.114473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2678.383608] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2678.391351] Lustre: Found index 0 for lustre-OST0000, updating log [ 2678.408327] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2678.537459] Lustre: lustre-OST0000: new disk, initializing [ 2678.542685] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2678.549413] Lustre: 101299:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2678.896032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2685.287512] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2688.692230] Lustre: Failing over lustre-OST0000 [ 2688.903447] Lustre: server umount lustre-OST0000 complete [ 2694.628800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2694.648570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2694.658297] Lustre: Skipped 1 previous similar message [ 2698.786245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2698.794961] Lustre: Skipped 1 previous similar message [ 2703.844643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2703.858294] Lustre: Skipped 1 previous similar message [ 2706.400206] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2706.626675] Lustre: server umount lustre-MDT0000 complete [ 2708.961225] LustreError: 98952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2708.979296] LustreError: 98952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2710.485718] LustreError: 98202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443445 with bad export cookie 6089462769865008963 [ 2710.504369] LustreError: 98202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2710.506256] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2710.879374] Lustre: server umount lustre-MDT0001 complete [ 2718.547884] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 09:50:51 (1788443451) [ 2732.859873] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 2743.362503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2743.794761] LustreError: 103646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2743.884143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2748.645918] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2757.007266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2761.341800] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2764.172714] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2770.654425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2771.437790] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2777.461669] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2782.707359] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2782.713040] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2782.731583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2785.301079] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2785.371702] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2785.477340] Lustre: lustre-OST0001: new disk, initializing [ 2785.482692] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2785.498928] Lustre: 105963:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2785.844569] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:56 [ 2785.864242] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2785.882845] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2785.887414] Lustre: Skipped 2 previous similar messages [ 2788.971067] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:03 [ 2788.992367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2789.000505] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2789.005351] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2789.009644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2792.693571] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2794.038617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2794.048368] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2794.127315] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2801.369668] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2808.229733] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2814.441823] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2814.460260] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2814.481615] Lustre: Skipped 1 previous similar message [ 2814.487930] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2820.489221] Lustre: server umount lustre-MDT0001 complete [ 2821.237687] LustreError: 103643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2821.273212] LustreError: 103643:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2830.928975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2831.072653] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2831.087084] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2831.145468] Lustre: 103636:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2832.972312] LustreError: 107544:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2832.978811] LustreError: 107544:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2832.985287] LustreError: 103639:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2833.080744] LustreError: 107550:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.147@tcp: cfg command failed: rc = -17 [ 2833.089925] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2833.089925] [ 2835.789831] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2836.999600] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2848.630078] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 09:53:00 (1788443580) [ 2871.217357] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 2889.745862] Lustre: 109581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2915.054875] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2919.759412] Lustre: 110717:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3020.774645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3020.787040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3020.808327] Lustre: Skipped 2 previous similar messages [ 3020.818113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3020.823653] Lustre: Skipped 6 previous similar messages [ 3024.726556] Lustre: server umount lustre-MDT0000 complete [ 3026.401749] LustreError: 103641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3026.446471] LustreError: 103641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 3029.612853] LustreError: 103628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443764 with bad export cookie 6089462769865009894 [ 3029.616816] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3029.620960] LustreError: 103628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3030.206885] Lustre: server umount lustre-MDT0001 complete [ 3046.960633] Lustre: server umount lustre-OST0000 complete [ 3047.713684] Lustre: 93235:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443766/real 1788443766] req@ffff9216f6a8dc00 x1875318598369024/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788443782 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3047.740111] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3047.756833] Lustre: Skipped 3 previous similar messages [ 3050.916997] Lustre: server umount lustre-OST0001 complete [ 3055.922547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3065.777390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3075.326794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3085.549608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3108.944226] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 3119.261226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3119.301634] Lustre: lustre-MDT0000: reset Object Index mappings [ 3120.023575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3120.029086] Lustre: Skipped 1 previous similar message [ 3124.818224] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3134.461898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3134.506895] Lustre: lustre-MDT0001: reset Object Index mappings [ 3140.300283] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3143.227954] Lustre: 115819:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3150.695235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3158.710833] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3159.665606] LustreError: 116172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3159.688735] LustreError: 116172:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3159.694410] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3164.803142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3167.814828] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3173.900062] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3173.911475] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3174.768082] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3182.913285] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3187.027377] Lustre: 117697:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3246.679316] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:59:38 (1788443978) [ 3271.145963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3271.157664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3271.172991] Lustre: Skipped 1 previous similar message [ 3271.205326] Lustre: Skipped 7 previous similar messages [ 3275.669664] Lustre: server umount lustre-MDT0000 complete [ 3282.886602] LustreError: 114650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444017 with bad export cookie 6089462769865056990 [ 3282.906517] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3282.908911] LustreError: 114650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3289.783320] Lustre: server umount lustre-MDT0001 complete [ 3302.003516] Lustre: server umount lustre-OST0000 complete [ 3307.886205] LustreError: 118663:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000401:0x75:0x0].0x0 (ffff9215c3033d00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3322.336679] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3322.610451] Lustre: server umount lustre-OST0001 complete [ 3330.636835] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 3340.798763] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 3386.247696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3386.648633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3386.721845] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3387.040317] Lustre: lustre-MDT0000: new disk, initializing [ 3387.212488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3387.215310] Lustre: Skipped 3 previous similar messages [ 3387.255795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3393.592108] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3403.418991] Lustre: server umount lustre-MDT0000 complete [ 3414.323075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3414.608835] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3414.626247] Lustre: 121989:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3414.635907] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3420.131971] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3426.006704] Lustre: server umount lustre-MDT0000 complete [ 3437.072404] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 10:02:50 (1788444170) [ 3439.108632] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3441.293040] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 10:02:54 (1788444174) [ 3464.992921] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 3477.284469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3482.444078] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3490.839756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3490.956903] Lustre: 124040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3491.043818] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3491.051364] Lustre: Skipped 1 previous similar message [ 3491.123254] Lustre: lustre-MDT0001: new disk, initializing [ 3491.220381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3491.226399] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3495.790422] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3500.541201] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3508.234982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3508.519531] Lustre: lustre-OST0000: new disk, initializing [ 3508.523863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3508.530903] Lustre: 125620:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3509.829818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3509.843310] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3510.055982] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3515.955765] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3526.636943] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3526.814751] Lustre: lustre-OST0001: new disk, initializing [ 3526.820829] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3526.830287] Lustre: 126491:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3528.905637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3528.922658] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3529.052241] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3533.486570] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3542.860283] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3546.405226] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3983.491739] Lustre: 132913:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3983.504692] Lustre: 132913:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3983.511468] Lustre: 132913:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3983.518105] Lustre: 132913:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3983.534275] Lustre: 132913:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3983.540698] Lustre: 132913:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4290.632907] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:17:03 (1788445023) [ 4310.497557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4310.509219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4310.531309] Lustre: Skipped 4 previous similar messages [ 4310.555190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4310.582108] Lustre: Skipped 4 previous similar messages [ 4312.561969] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4314.071730] Lustre: server umount lustre-MDT0000 complete [ 4317.670965] LustreError: 124048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4317.733224] LustreError: 124048:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 4319.681223] LustreError: 132954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445054 with bad export cookie 6089462769865126864 [ 4319.683430] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4319.704148] LustreError: 132954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4322.796339] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4322.816046] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4322.827327] Lustre: Skipped 3 previous similar messages [ 4322.851287] Lustre: Skipped 5 previous similar messages [ 4326.314153] Lustre: server umount lustre-MDT0001 complete [ 4336.665869] Lustre: server umount lustre-OST0000 complete [ 4347.265623] Lustre: server umount lustre-OST0001 complete [ 4354.533028] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 4364.704956] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 4407.180057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4407.515408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4407.573731] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4407.660253] Lustre: lustre-MDT0000: new disk, initializing [ 4407.732764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4407.737473] Lustre: Skipped 5 previous similar messages [ 4407.753094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4413.932773] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4426.056703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4426.233989] Lustre: 135983:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4426.294872] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4426.298749] Lustre: Skipped 1 previous similar message [ 4426.389519] Lustre: lustre-MDT0001: new disk, initializing [ 4426.533508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4426.552663] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4431.217920] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4441.929699] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4449.809608] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4457.916881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4458.324794] Lustre: lustre-OST0000: new disk, initializing [ 4458.331211] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4458.342177] Lustre: 137953:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4458.418407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4458.425947] Lustre: Skipped 1 previous similar message [ 4460.172349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4460.197762] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4460.403938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4466.162193] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4476.794234] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4487.789416] Lustre: lustre-MDD0000: changelog on [ 4490.957460] Lustre: lustre-MDD0001: changelog on [ 6619.561193] Lustre: lustre-MDD0001: changelog off [ 6622.049030] Lustre: 137312:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6623.335620] Lustre: lustre-MDD0000: changelog off [ 6634.499431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6634.517848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6634.518028] Lustre: Skipped 1 previous similar message [ 6634.542302] Lustre: Skipped 3 previous similar messages [ 6638.550491] Lustre: server umount lustre-OST0000 complete [ 6643.681327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6643.686966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6643.698222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6648.725222] Lustre: server umount lustre-MDT0000 complete [ 6649.838801] LustreError: 136861:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6649.858816] LustreError: 136861:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6652.434776] LustreError: 135975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447387 with bad export cookie 6089462769865903913 [ 6652.452168] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6652.455763] LustreError: 135975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6652.841177] Lustre: server umount lustre-MDT0001 complete [ 6663.957583] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 6667.050692] Key type lgssc unregistered [ 6667.420646] LNet: 147143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6667.426749] LNetError: 147143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6667.448768] LNet: Removed LNI 192.168.203.147@tcp [ 6668.565056] Key type .llcrypt unregistered [ 6668.574096] Key type ._llcrypt unregistered [ 6689.702099] Key type ._llcrypt registered [ 6689.704145] Key type .llcrypt registered [ 6689.850805] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 6703.355301] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 6704.084811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6704.263542] alg: No test for adler32 (adler32-zlib) [ 6705.332720] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6705.622544] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 6707.368167] Key type lgssc registered [ 6708.824822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6748.670418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6748.692160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6750.009345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6750.058563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6750.135088] Lustre: lustre-MDT0000: new disk, initializing [ 6750.241344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6750.260179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6754.606323] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6765.146556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6765.306727] Lustre: 150338:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6765.376745] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6765.381418] Lustre: Skipped 1 previous similar message [ 6765.457772] Lustre: lustre-MDT0001: new disk, initializing [ 6765.637886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6765.698685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6765.717701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6769.451631] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6779.981864] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6786.742769] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6796.327327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6796.772553] Lustre: lustre-OST0000: new disk, initializing [ 6796.775747] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6796.781061] Lustre: 152307:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6796.857436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6804.440985] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6805.622793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6805.633101] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6805.839455] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6814.193661] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6820.739026] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_module obdecho/obdecho [ 6822.728356] LustreError: 153312:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6822.734866] LustreError: 153312:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6824.525877] LustreError: 153408:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6836.722871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6836.728976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6836.741038] Lustre: Skipped 1 previous similar message [ 6836.753556] Lustre: Skipped 2 previous similar messages [ 6841.828306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6841.839679] Lustre: Skipped 2 previous similar messages [ 6843.235567] Lustre: server umount lustre-MDT0000 complete [ 6847.075322] LustreError: 150330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447581 with bad export cookie 17955702680131812747 [ 6847.079130] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6847.090247] LustreError: 150330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6847.433454] Lustre: server umount lustre-MDT0001 complete [ 6860.495865] Lustre: server umount lustre-OST0000 complete [ 6881.120948] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 6890.769594] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 6929.655974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6929.940396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6929.968806] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6930.038397] Lustre: lustre-MDT0000: new disk, initializing [ 6930.101777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6930.121905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6933.755636] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6944.202791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6944.317744] Lustre: 157135:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6944.385055] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6944.388148] Lustre: Skipped 1 previous similar message [ 6944.457972] Lustre: lustre-MDT0001: new disk, initializing [ 6944.546072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6944.583283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6944.600782] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6949.755934] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6960.482921] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6966.460832] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6973.528676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6973.886095] Lustre: lustre-OST0000: new disk, initializing [ 6973.889763] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6973.894506] Lustre: 159105:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6974.010297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6980.380286] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6982.183949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6982.192490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6982.266230] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6989.865676] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6994.067863] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6994.244771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6997.202304] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6997.367579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7002.598068] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7002.599845] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7002.614795] Lustre: Skipped 1 previous similar message [ 7002.638291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7002.645770] Lustre: Skipped 1 previous similar message [ 7007.716209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7007.729766] Lustre: Skipped 1 previous similar message [ 7012.841724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7012.864622] Lustre: Skipped 1 previous similar message [ 7013.349533] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7013.525877] Lustre: server umount lustre-OST0000 complete [ 7017.961385] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7017.974691] Lustre: Skipped 2 previous similar messages [ 7023.073458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7023.078837] Lustre: Skipped 3 previous similar messages [ 7023.525193] Lustre: server umount lustre-MDT0000 complete [ 7026.662840] LustreError: 157127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447761 with bad export cookie 17955702680131813706 [ 7026.672752] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7026.949568] Lustre: server umount lustre-MDT0001 complete [ 7030.070181] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 11:02:43 (1788447763) [ 7031.527949] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7033.136582] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 11:02:46 (1788447766) [ 7041.728888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7042.111101] LustreError: 161521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7042.128981] LustreError: 161521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7042.188438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7047.649582] LustreError: 161522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7048.042185] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7051.755020] LustreError: 161521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7056.889807] LustreError: 161522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7059.507509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7063.640599] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7070.763538] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7075.664744] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7081.489164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7081.800417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7081.808096] Lustre: Skipped 1 previous similar message [ 7086.886866] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7092.527958] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7098.513173] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.47@tcp (stopping) [ 7100.901624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7100.914138] LustreError: Skipped 1 previous similar message [ 7100.920567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7104.300981] Lustre: server umount lustre-MDT0000 complete [ 7107.562972] LustreError: 163269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7107.610577] LustreError: 163269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7108.441864] LustreError: 161503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447843 with bad export cookie 17955702680131814588 [ 7108.444935] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7108.458399] LustreError: 161503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7112.688825] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7112.710993] Lustre: Skipped 2 previous similar messages [ 7114.907298] Lustre: server umount lustre-MDT0001 complete [ 7121.828398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7135.274689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7135.517589] LustreError: 165360:0:(llog.c:1655:llog_backup()) MGC192.168.203.147@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7135.528569] Lustre: 165360:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.147@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7138.274956] LustreError: 165366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7138.302963] LustreError: 165366:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7154.786724] LustreError: 165367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7154.810977] LustreError: 165367:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 7189.474350] LustreError: 163353:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7189.500803] LustreError: 163353:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [ 7196.128322] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7196.132375] Lustre: 165360:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.147@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7237.107136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7241.538724] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7242.217259] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7250.950555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7253.478907] LustreError: 165392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7253.490069] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf92f787445c95df2 [ 7253.513827] LustreError: 165392:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 45 previous similar messages [ 7253.553339] Lustre: MGC192.168.203.147@tcp: Connection restored to 0@lo (at 0@lo) [ 7259.029397] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7259.118669] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7271.490783] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7274.525465] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7281.353866] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7283.150383] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7305.186360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7305.191132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7305.205498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7305.205506] Lustre: Skipped 9 previous similar messages [ 7305.239056] Lustre: Skipped 1 previous similar message [ 7306.130169] Lustre: server umount lustre-MDT0000 complete [ 7309.374700] LustreError: 161503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448044 with bad export cookie 17955702680131821042 [ 7309.384426] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7309.694720] Lustre: server umount lustre-MDT0001 complete [ 7322.648638] Lustre: server umount lustre-OST0000 complete [ 7339.513883] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 7348.543632] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 7388.036114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7388.275191] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7388.299798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7388.408879] Lustre: lustre-MDT0000: new disk, initializing [ 7388.497271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7388.502705] Lustre: Skipped 1 previous similar message [ 7388.514171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7393.036457] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7406.081094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7406.188680] Lustre: 170469:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7406.415887] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7411.106845] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7422.389868] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7427.965818] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7435.466829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7435.666375] Lustre: lustre-OST0000: new disk, initializing [ 7435.669430] Lustre: Skipped 1 previous similar message [ 7435.673443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7435.678866] Lustre: Skipped 2 previous similar messages [ 7435.686983] Lustre: 172437:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7442.393454] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7445.077060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7445.092975] Lustre: Skipped 1 previous similar message [ 7445.106459] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7445.181581] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7454.578666] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7461.026821] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7461.354696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7467.541256] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7467.750562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7471.080348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7471.084644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7471.090409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7471.090418] Lustre: Skipped 2 previous similar messages [ 7471.156819] Lustre: Skipped 2 previous similar messages [ 7476.699687] Lustre: server umount lustre-OST0000 complete [ 7487.365232] Lustre: server umount lustre-MDT0000 complete [ 7490.995389] LustreError: 170461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448225 with bad export cookie 17955702680131837877 [ 7491.009672] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7491.015500] LustreError: 170461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7491.459942] Lustre: server umount lustre-MDT0001 complete [ 7501.183311] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:10:34 (1788448234) [ 7508.428036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7509.076691] LustreError: 174850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7514.457731] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7523.433347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7529.274991] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7537.215490] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7543.722152] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7550.447565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7557.663612] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7565.619602] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7581.268986] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:11:53 (1788448313) [ 7589.860557] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7589.862683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7589.865470] LustreError: Skipped 1 previous similar message [ 7589.878410] Lustre: Skipped 3 previous similar messages [ 7591.647956] Lustre: server umount lustre-OST0000 complete [ 7600.108984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7600.117538] Lustre: Skipped 10 previous similar messages [ 7610.337212] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7610.548386] Lustre: server umount lustre-MDT0000 complete [ 7615.073064] LustreError: 177627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448349 with bad export cookie 17955702680131838759 [ 7615.082827] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7615.094142] LustreError: 177627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7615.865343] Lustre: server umount lustre-MDT0001 complete [ 7629.504762] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 7632.305200] Key type lgssc unregistered [ 7632.587690] LNet: 178586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7632.602928] LNetError: 178586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7632.620913] LNet: Removed LNI 192.168.203.147@tcp [ 7633.625200] Key type .llcrypt unregistered [ 7633.630995] Key type ._llcrypt unregistered [ 7653.347954] Key type ._llcrypt registered [ 7653.353781] Key type .llcrypt registered [ 7653.439121] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 7654.105236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7654.246199] alg: No test for adler32 (adler32-zlib) [ 7655.200876] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7655.383589] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 7657.040164] Key type lgssc registered [ 7657.990083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7667.241180] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7667.254257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7682.912368] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7688.032947] LustreError: 179831:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.147@tcp: failed processing log, type 4: rc = -110 [ 7693.216639] LustreError: 179831:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7693.238127] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7693.244825] LustreError: MGC192.168.203.147@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7693.262948] LustreError: 179831:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7693.282598] LustreError: 179831:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7693.295819] LustreError: 179831:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7693.433472] Lustre: server umount lustre-OST0000 complete [ 7693.437328] LustreError: 179831:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7702.972933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7704.486277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7709.361569] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7714.790345] LustreError: 180330:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7714.815508] LustreError: 180330:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7718.961112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7719.395876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7724.300290] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7731.971490] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7739.506225] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7747.880600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7748.093839] LustreError: 182149:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7748.109312] Lustre: 182149:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.147@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7748.277109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7756.800452] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7766.459566] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7770.088028] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7770.093554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7770.096241] Lustre: Skipped 1 previous similar message [ 7770.105375] Lustre: Skipped 2 previous similar messages [ 7775.204292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7775.215944] Lustre: Skipped 2 previous similar messages [ 7780.328531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7780.333201] Lustre: Skipped 2 previous similar messages [ 7783.392355] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7783.718904] Lustre: server umount lustre-MDT0000 complete [ 7785.461595] LustreError: 180331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7785.485074] LustreError: 180331:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7787.937630] LustreError: 181066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448522 with bad export cookie 3547198262899423732 [ 7787.938904] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7787.951508] LustreError: 181066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7788.228785] Lustre: server umount lustre-MDT0001 complete [ 7803.050370] Lustre: server umount lustre-OST0000 complete [ 7813.107670] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 7823.692136] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 7862.560865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7862.910777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7862.939300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7863.044239] Lustre: lustre-MDT0000: new disk, initializing [ 7863.138720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7863.167513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7867.293716] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7877.358201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7877.501080] Lustre: 185677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7877.574359] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7877.580138] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7877.586163] Lustre: Skipped 1 previous similar message [ 7877.655708] Lustre: lustre-MDT0001: new disk, initializing [ 7877.769500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7877.812620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7877.832674] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7882.477385] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7891.507193] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7896.913689] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7903.528302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7903.808883] Lustre: lustre-OST0000: new disk, initializing [ 7903.814471] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7903.822071] Lustre: 187646:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7903.907462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7910.380832] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7910.964426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7910.977078] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7911.098744] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7920.183196] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7925.038907] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7925.229538] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7929.705551] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7929.964370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7931.873765] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7931.880021] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7931.887927] LustreError: Skipped 1 previous similar message [ 7931.896242] Lustre: Skipped 1 previous similar message [ 7931.914036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7940.068769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7940.075970] Lustre: Skipped 4 previous similar messages [ 7946.208576] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7946.477465] Lustre: server umount lustre-OST0000 complete [ 7954.403115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7954.409340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7956.906111] Lustre: server umount lustre-MDT0000 complete [ 7960.241078] LustreError: 186471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448694 with bad export cookie 3547198262899424453 [ 7960.249934] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7960.254117] LustreError: 186471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7960.565738] Lustre: server umount lustre-MDT0001 complete [ 7969.992098] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:18:23 (1788448703) [ 7987.363640] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 7989.873357] Key type lgssc unregistered [ 7990.171379] LNet: 190598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7990.183406] LNetError: 190598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7990.211996] LNet: Removed LNI 192.168.203.147@tcp [ 7991.259169] Key type .llcrypt unregistered [ 7991.261536] Key type ._llcrypt unregistered [ 8012.429846] Key type ._llcrypt registered [ 8012.432199] Key type .llcrypt registered [ 8012.509718] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8013.931509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8014.050934] alg: No test for adler32 (adler32-zlib) [ 8015.034787] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8015.339436] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 8017.040198] Key type lgssc registered [ 8018.229944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8026.175388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8026.206690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8027.564404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8031.937538] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8037.865779] LustreError: 191815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8037.896236] LustreError: 191815:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8041.021990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8041.372592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8045.740359] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8051.981487] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8056.879059] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8063.116883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8063.313904] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8063.317712] LustreError: 193677:0:(llog.c:1696:llog_backup()) MGC192.168.203.147@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8063.329787] Lustre: 193677:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.147@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8063.473674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8068.716982] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8075.640870] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8077.799132] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8077.801575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8077.817271] Lustre: Skipped 1 previous similar message [ 8077.828234] Lustre: Skipped 1 previous similar message [ 8082.403274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8082.424789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8082.447254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8083.939768] Lustre: server umount lustre-MDT0000 complete [ 8087.803340] LustreError: 191796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448822 with bad export cookie 15410952897463927408 [ 8087.814923] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8088.066963] Lustre: server umount lustre-MDT0001 complete [ 8102.416621] Lustre: server umount lustre-OST0000 complete [ 8111.784281] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 8120.929659] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8153.235906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8153.429640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8153.451503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8153.530082] Lustre: lustre-MDT0000: new disk, initializing [ 8153.606225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8153.638361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8158.134936] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8167.972151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8168.061141] Lustre: 197201:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8168.102527] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8168.105260] Lustre: Skipped 1 previous similar message [ 8168.108268] LustreError: 197982:0:(llog.c:1696:llog_backup()) MGC192.168.203.147@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8168.113958] LustreError: 197982:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8168.118866] Lustre: 197982:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.147@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8168.139318] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8168.144611] Lustre: Skipped 1 previous similar message [ 8168.240732] Lustre: lustre-MDT0001: new disk, initializing [ 8168.343563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8168.378156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8168.396242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8172.592794] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8182.275778] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8187.317811] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8194.563857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8194.915835] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8194.922705] Lustre: Skipped 1 previous similar message [ 8194.932653] LustreError: 199145:0:(llog.c:1696:llog_backup()) MGC192.168.203.147@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8194.947526] LustreError: 199145:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8194.960395] Lustre: 199145:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.147@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8194.977306] Lustre: 199145:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8195.020929] Lustre: lustre-OST0000: new disk, initializing [ 8195.027982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8195.036433] Lustre: 199168:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8195.137345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8201.457438] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8203.335814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8203.340795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8203.446803] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8210.498481] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8215.580713] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8215.837645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8221.036775] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8221.331242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8224.225153] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8224.228672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8224.239031] Lustre: Skipped 1 previous similar message [ 8224.239153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8224.248138] Lustre: Skipped 3 previous similar messages [ 8229.327365] Lustre: server umount lustre-OST0000 complete [ 8234.473199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8234.484452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8234.514964] Lustre: Skipped 1 previous similar message [ 8234.525926] Lustre: Skipped 1 previous similar message [ 8239.586902] Lustre: server umount lustre-MDT0000 complete [ 8239.591743] LustreError: 197212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8239.619299] LustreError: 197212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8244.711150] LustreError: 197211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8244.748339] LustreError: 197211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8244.861188] LustreError: 197194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448979 with bad export cookie 15410952897463928122 [ 8244.878433] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8244.890444] LustreError: 197194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8245.494719] Lustre: server umount lustre-MDT0001 complete [ 8255.893373] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:23:08 (1788448988) [ 8273.004669] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8283.275892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8283.753086] LustreError: 202297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8283.823417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8288.016707] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8289.259644] LustreError: 202298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8296.923366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8302.612266] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8306.002638] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8312.706785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8319.521420] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8331.557980] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8331.779574] Lustre: lustre-OST0001: new disk, initializing [ 8331.788288] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8331.802301] Lustre: 204607:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8331.910354] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8331.916636] Lustre: Skipped 2 previous similar messages [ 8339.211490] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8340.201152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8340.230237] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8340.455727] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8352.362828] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8360.879793] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8363.173954] Lustre: DEBUG MARKER: ADD OST3 [ 8369.161250] loop: module loaded [ 8369.313783] loop0: detected capacity change from 0 to 204800000 [ 8369.717949] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8369.756856] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8369.784314] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8377.452496] loop0: detected capacity change from 0 to 204800000 [ 8377.813446] Lustre: DEBUG MARKER: STOP OST3 [ 8385.265797] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8385.631178] Lustre: lustre-OST0003: new disk, initializing [ 8385.635926] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8385.646995] Lustre: 206641:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8387.145702] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8387.316670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8387.327767] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8387.381533] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8389.154140] LustreError: 206753:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8389.252343] Lustre: Failing over lustre-OST0003 [ 8389.375241] Lustre: server umount lustre-OST0003 complete [ 8392.344378] LustreError: 203784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8392.360606] LustreError: 203784:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8392.677198] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8392.692637] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8392.711442] Lustre: Skipped 1 previous similar message [ 8393.081807] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8401.163651] Lustre: DEBUG MARKER: START OST3 again [ 8408.351044] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8408.646043] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8408.657618] Lustre: Skipped 1 previous similar message [ 8408.675320] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8409.794944] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8410.580372] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8410.581276] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8415.359437] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8437.246903] Lustre: Failing over lustre-OST0003 [ 8437.567515] Lustre: server umount lustre-OST0003 complete [ 8439.296612] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8439.302711] LustreError: 203786:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8439.318065] Lustre: Skipped 1 previous similar message [ 8439.367300] LustreError: 203786:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 8456.162057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8456.176401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8456.197684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8462.360964] Lustre: server umount lustre-MDT0000 complete [ 8465.889255] LustreError: 205375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449200 with bad export cookie 15410952897463929011 [ 8465.903162] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8465.906601] LustreError: 205375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8469.989219] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8469.999543] Lustre: Skipped 4 previous similar messages [ 8472.464723] Lustre: server umount lustre-MDT0001 complete [ 8476.754069] Lustre: server umount lustre-OST0000 complete [ 8480.966558] Lustre: server umount lustre-OST0001 complete [ 8488.007932] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 8497.086335] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8537.157183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8537.545843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8537.606600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8537.718804] Lustre: lustre-MDT0000: new disk, initializing [ 8537.884225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8537.936942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8543.534043] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8554.665592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8554.860738] Lustre: 212067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8554.943258] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8554.946483] Lustre: Skipped 1 previous similar message [ 8555.028880] Lustre: lustre-MDT0001: new disk, initializing [ 8555.196469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8555.211447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8559.940569] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8568.627543] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8573.975599] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8581.037505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8581.378472] Lustre: 214037:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8583.197294] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8583.391722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8588.368667] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8597.564504] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8602.213187] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8602.474305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8606.700947] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8606.874297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8608.740282] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8608.746576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8608.772053] Lustre: Skipped 5 previous similar messages [ 8608.778917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8608.791421] Lustre: Skipped 3 previous similar messages [ 8614.768753] Lustre: server umount lustre-OST0000 complete [ 8624.714990] Lustre: server umount lustre-MDT0000 complete [ 8628.235689] LustreError: 212059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449362 with bad export cookie 15410952897463931307 [ 8628.239884] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8628.249642] LustreError: 212059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8628.613179] Lustre: server umount lustre-MDT0001 complete [ 8631.814621] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:29:25 (1788449365) [ 8648.282164] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 8658.723801] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8700.750572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8700.980694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8701.021439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8701.024479] Lustre: Skipped 1 previous similar message [ 8701.079879] Lustre: lustre-MDT0000: new disk, initializing [ 8701.082631] Lustre: Skipped 1 previous similar message [ 8701.157428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8701.162471] Lustre: Skipped 1 previous similar message [ 8705.726853] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8717.467057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8717.745764] Lustre: 218649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8718.298125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8724.629205] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8735.106451] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8741.422862] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8748.475366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8748.837374] Lustre: 220617:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8750.977563] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8751.211237] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8756.493437] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8766.799405] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8771.921980] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8772.159221] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8777.023768] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8777.252267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8781.793921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8781.801872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8781.812765] LustreError: Skipped 2 previous similar messages [ 8781.814928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8781.814936] Lustre: Skipped 8 previous similar messages [ 8781.863748] Lustre: Skipped 4 previous similar messages [ 8785.546176] Lustre: server umount lustre-OST0000 complete [ 8795.912311] Lustre: server umount lustre-MDT0000 complete [ 8797.154593] LustreError: 218652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8797.175542] LustreError: 218652:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 8799.648868] LustreError: 218641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449534 with bad export cookie 15410952897463932189 [ 8799.661901] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8799.669326] LustreError: 218641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8800.078657] Lustre: server umount lustre-MDT0001 complete [ 8808.614560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8809.193456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8809.208353] Lustre: Skipped 5 previous similar messages [ 8813.718631] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8822.684751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8827.731419] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8836.115316] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8841.969910] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8848.410714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8854.320818] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8861.777918] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8873.446382] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8873.459197] LustreError: Skipped 1 previous similar message [ 8873.460882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8873.480847] Lustre: Skipped 5 previous similar messages [ 8874.325305] Lustre: server umount lustre-OST0000 complete [ 8892.384151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8892.600902] Lustre: server umount lustre-MDT0000 complete [ 8893.921450] LustreError: 222833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8893.947408] LustreError: 222833:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8897.325540] LustreError: 225709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449632 with bad export cookie 15410952897463933071 [ 8897.340344] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8897.362563] LustreError: 225709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8897.878031] Lustre: server umount lustre-MDT0001 complete [ 9001.724760] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 9004.340319] Key type lgssc unregistered [ 9004.642816] LNet: 226499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9004.656224] LNetError: 226499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9004.675701] LNet: Removed LNI 192.168.203.147@tcp [ 9005.757156] Key type .llcrypt unregistered [ 9005.759621] Key type ._llcrypt unregistered [ 9034.847968] Key type ._llcrypt registered [ 9034.853223] Key type .llcrypt registered [ 9035.005686] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 9051.835357] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 9053.374452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9053.501331] alg: No test for adler32 (adler32-zlib) [ 9054.671639] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9054.990429] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 9056.649115] Key type lgssc registered [ 9057.819192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9094.072240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9094.096914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9095.265219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9095.332807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9095.421176] Lustre: lustre-MDT0000: new disk, initializing [ 9095.540189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9095.560716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9101.039891] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9113.996531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9114.251289] Lustre: 229855:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9114.353985] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9114.367060] Lustre: Skipped 1 previous similar message [ 9114.524129] Lustre: lustre-MDT0001: new disk, initializing [ 9114.617438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9114.649921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9114.665853] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9119.808760] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9128.877766] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9135.134944] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9142.624737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9143.003796] Lustre: lustre-OST0000: new disk, initializing [ 9143.008941] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9143.018888] Lustre: 231826:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9143.148824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9149.545219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9149.558492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9149.739966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9150.140843] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9159.370909] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9164.821808] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9165.084554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9171.539226] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9171.873737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9175.527289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9175.528635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9175.528709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9175.553195] Lustre: Skipped 1 previous similar message [ 9180.590095] Lustre: server umount lustre-OST0000 complete [ 9185.767623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9185.770664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9185.784910] Lustre: Skipped 1 previous similar message [ 9185.800816] Lustre: Skipped 2 previous similar messages [ 9190.883743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9190.896853] Lustre: Skipped 1 previous similar message [ 9191.180093] Lustre: server umount lustre-MDT0000 complete [ 9195.403115] LustreError: 229847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449930 with bad export cookie 1452733464751686247 [ 9195.415393] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9195.421933] LustreError: 229847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9195.860740] Lustre: server umount lustre-MDT0001 complete [ 9217.087666] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 9227.982105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9228.659520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9233.835768] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9233.896390] LustreError: 234760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9233.913199] LustreError: 234760:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9239.015044] LustreError: 234760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9243.747877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9244.143750] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9244.272157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9249.519765] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9253.347426] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9260.950057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9261.517832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9269.294408] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9278.692752] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9278.872234] Lustre: lustre-OST0001: new disk, initializing [ 9278.878801] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9278.884495] Lustre: 237068:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9278.981224] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9285.269773] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9288.796187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9288.804693] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9288.926865] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9296.129751] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9302.244724] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9315.486567] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:40:48 (1788450048) [ 9319.916242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9319.944585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9319.958970] Lustre: Skipped 2 previous similar messages [ 9330.147281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9330.157766] Lustre: Skipped 7 previous similar messages [ 9332.192171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9332.338187] Lustre: server umount lustre-MDT0000 complete [ 9335.267162] LustreError: 234760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9335.277204] LustreError: 234760:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9336.093965] LustreError: 237835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450070 with bad export cookie 1452733464751687129 [ 9336.099541] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9336.107915] LustreError: 237835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9336.310819] Lustre: server umount lustre-MDT0001 complete [ 9349.703756] Lustre: server umount lustre-OST0000 complete [ 9364.338480] Lustre: server umount lustre-OST0001 complete [ 9372.881267] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 9383.714885] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 9423.420986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9423.669465] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9423.695876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9423.785748] Lustre: lustre-MDT0000: new disk, initializing [ 9423.882196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9423.901395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9428.875237] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9440.891327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9441.029269] Lustre: 241016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9441.093582] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9441.098574] Lustre: Skipped 1 previous similar message [ 9441.209735] Lustre: lustre-MDT0001: new disk, initializing [ 9441.394546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9441.414834] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9446.530273] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9457.609415] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9464.232339] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9471.466650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9471.793554] Lustre: lustre-OST0000: new disk, initializing [ 9471.797098] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9471.802530] Lustre: 242988:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9472.981328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9472.999478] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9473.161261] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9478.127062] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9488.136356] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9493.640379] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9493.863480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9498.756514] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9498.995125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9503.720296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9503.734725] Lustre: Skipped 3 previous similar messages [ 9503.740687] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9503.751471] LustreError: Skipped 1 previous similar message [ 9503.751768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9503.763530] Lustre: Skipped 1 previous similar message [ 9507.228289] Lustre: server umount lustre-OST0000 complete [ 9512.930602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9512.943210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9512.968956] Lustre: Skipped 1 previous similar message [ 9517.856568] Lustre: server umount lustre-MDT0000 complete [ 9519.081510] LustreError: 241026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9519.104058] LustreError: 241026:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9521.760784] LustreError: 241009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450256 with bad export cookie 1452733464751689257 [ 9521.768549] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9521.777481] LustreError: 241009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9522.237896] Lustre: server umount lustre-MDT0001 complete [ 9531.328938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9531.908761] LustreError: 245203:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9531.975892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9531.981966] Lustre: Skipped 2 previous similar messages [ 9536.961862] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9542.117271] LustreError: 245204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9542.133249] LustreError: 245204:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9545.724412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9550.852030] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9558.645383] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9564.210745] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9570.929918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9577.295391] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9584.822700] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9730.018794] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9730.029175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9730.044340] Lustre: Skipped 1 previous similar message [ 9730.048746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9730.056963] Lustre: Skipped 3 previous similar messages [ 9734.126565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9744.354903] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9744.543567] Lustre: server umount lustre-OST0000 complete [ 9749.486490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9749.506271] Lustre: Skipped 1 previous similar message [ 9762.273243] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9762.373928] Lustre: server umount lustre-MDT0000 complete [ 9764.833720] LustreError: 245931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9764.844861] LustreError: 245931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9766.195750] LustreError: 245185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450500 with bad export cookie 1452733464751690146 [ 9766.200789] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9766.211055] LustreError: 245185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9766.557862] Lustre: server umount lustre-MDT0001 complete [ 9854.128213] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 9856.771663] Key type lgssc unregistered [ 9857.046575] LNet: 248813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9857.056245] LNetError: 248813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9857.075694] LNet: Removed LNI 192.168.203.147@tcp [ 9857.861652] Key type .llcrypt unregistered [ 9857.865089] Key type ._llcrypt unregistered [ 9884.463295] Key type ._llcrypt registered [ 9884.464942] Key type .llcrypt registered [ 9884.592268] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 9901.331793] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 9902.618468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9902.700231] alg: No test for adler32 (adler32-zlib) [ 9903.810793] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9904.113492] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 9905.848157] Key type lgssc registered [ 9907.031566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9955.343197] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 9966.514991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9966.538361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9967.825844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9967.866683] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9968.025652] Lustre: lustre-MDT0000: new disk, initializing [ 9968.103473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9968.119645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9972.805781] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9985.288457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9985.468790] Lustre: 252888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9985.532311] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9985.535508] Lustre: Skipped 1 previous similar message [ 9985.637578] Lustre: lustre-MDT0001: new disk, initializing [ 9985.790434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9985.846384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9985.873350] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9990.751881] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9996.035827] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10003.672918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10004.074861] Lustre: lustre-OST0000: new disk, initializing [10004.079140] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10004.085234] Lustre: 254518:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10004.207156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10010.190047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10010.198637] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10010.329213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10011.116348] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10022.675915] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10022.885609] Lustre: lustre-OST0001: new disk, initializing [10022.890960] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10022.908498] Lustre: 255390:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10023.052962] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10029.696616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10029.715588] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10029.831284] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10029.917507] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10041.461542] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10052.204052] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10063.410278] Lustre: server umount lustre-MDT0000 complete [10065.902316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10065.914105] Lustre: Skipped 3 previous similar messages [10066.492177] LustreError: 253682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450801 with bad export cookie 15031984086184695680 [10066.502633] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10066.513198] LustreError: 253682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10068.948275] Lustre: server umount lustre-MDT0001 complete [10082.691275] Lustre: server umount lustre-OST0000 complete [10096.192370] Lustre: server umount lustre-OST0001 complete [10108.695911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10116.440675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10123.999696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10132.473274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10151.430226] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [10159.872322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10159.900378] Lustre: lustre-MDT0000: reset Object Index mappings [10160.275800] LustreError: 260400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10160.295168] LustreError: 260400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10160.375282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10164.375481] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10165.733700] LustreError: 260401:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10170.858784] LustreError: 260400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10171.729660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10171.764212] Lustre: lustre-MDT0001: reset Object Index mappings [10172.250766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10177.208409] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10180.039906] Lustre: 261544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10187.913599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10188.251507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10196.422933] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10196.457500] LustreError: 261896:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10201.572397] LustreError: 261896:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10204.927320] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10206.218943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10206.221050] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10210.625866] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10217.768910] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10221.084969] Lustre: 263417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10226.662707] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10226.676140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10226.683087] Lustre: Skipped 2 previous similar messages [10226.690513] Lustre: Skipped 1 previous similar message [10227.895339] Lustre: server umount lustre-MDT0000 complete [10231.274792] LustreError: 260383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450966 with bad export cookie 15031984086184702470 [10231.277921] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10231.290805] LustreError: 260383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10231.581654] Lustre: server umount lustre-MDT0001 complete [10245.255134] Lustre: server umount lustre-OST0000 complete [10248.164142] Lustre: 250391:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450966/real 1788450966] req@ffff9215c78e9180 x1875326416318976/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788450982 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10248.204717] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10248.223164] Lustre: Skipped 1 previous similar message [10248.694715] Lustre: server umount lustre-OST0001 complete [10266.439855] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:56:39 (1788450999) [10284.273693] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [10296.975248] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [10333.562746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10333.803718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10333.851350] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10333.934460] Lustre: lustre-MDT0000: new disk, initializing [10334.012887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10334.021882] Lustre: Skipped 1 previous similar message [10334.046528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10338.979189] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10350.124590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10350.286971] Lustre: 267668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10350.392263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10350.399429] Lustre: Skipped 1 previous similar message [10350.486525] Lustre: lustre-MDT0001: new disk, initializing [10350.634834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10350.642817] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10354.966470] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10363.704610] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10368.412861] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10374.387399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10374.644573] Lustre: lustre-OST0000: new disk, initializing [10374.651958] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10374.665184] Lustre: 269637:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10376.637960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10376.654587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10376.778407] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10380.571398] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10389.287426] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10394.136195] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10394.314227] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10398.713410] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10398.929536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10402.281527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10402.283422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10402.290291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10402.297530] LustreError: Skipped 1 previous similar message [10402.322636] Lustre: Skipped 1 previous similar message [10407.239503] Lustre: server umount lustre-OST0000 complete [10412.002526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10412.017136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10412.032294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10412.040522] Lustre: Skipped 1 previous similar message [10416.929199] Lustre: server umount lustre-MDT0000 complete [10417.635873] LustreError: 268551:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10417.653614] LustreError: 268551:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10420.879217] LustreError: 267662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451155 with bad export cookie 15031984086184704563 [10420.892128] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10420.899791] LustreError: 267662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10421.433251] Lustre: server umount lustre-MDT0001 complete [10441.089217] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [10452.047326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10452.510295] LustreError: 272571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10452.557932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10452.566095] Lustre: Skipped 2 previous similar messages [10457.034818] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10465.539809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10469.463494] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10472.552429] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10479.113868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10485.983635] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10494.566636] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10494.745738] Lustre: lustre-OST0001: new disk, initializing [10494.749052] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10494.754807] Lustre: 274879:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10496.137148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10496.153200] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10496.257370] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10501.584269] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10511.067145] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10514.667844] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10519.539520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10519.644864] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10527.201215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10527.212393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10527.239281] Lustre: Skipped 1 previous similar message [10527.249606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10527.258912] Lustre: Skipped 1 previous similar message [10532.782402] Lustre: server umount lustre-MDT0000 complete [10534.372105] LustreError: 273302:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10534.394865] LustreError: 273302:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10536.280088] LustreError: 273307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451271 with bad export cookie 15031984086184705445 [10536.285837] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10536.300084] LustreError: 273307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10536.767806] Lustre: server umount lustre-MDT0001 complete [10550.877841] Lustre: server umount lustre-OST0000 complete [10564.899285] Lustre: server umount lustre-OST0001 complete [10580.932691] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [10591.620435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10592.311159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10592.320301] Lustre: Skipped 3 previous similar messages [10596.804563] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10601.445543] LustreError: 277956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10601.468410] LustreError: 277956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10605.969872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10611.462177] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10614.365404] Lustre: 279097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10620.744904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10626.619403] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10635.295320] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10636.678182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10636.713819] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10642.797198] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10650.832766] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10654.574422] Lustre: 280972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10659.786314] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10659.859970] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10670.143581] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 12:03:23 (1788451403) [10687.986829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10687.997229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10688.010135] Lustre: Skipped 6 previous similar messages [10688.022985] Lustre: Skipped 7 previous similar messages [10692.370383] Lustre: server umount lustre-MDT0000 complete [10696.130474] LustreError: 277936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451430 with bad export cookie 15031984086184712207 [10696.137786] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10696.146159] LustreError: 277936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10698.212634] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10698.219812] Lustre: Skipped 1 previous similar message [10702.991094] Lustre: server umount lustre-MDT0001 complete [10722.272625] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10722.322850] Lustre: server umount lustre-OST0000 complete [10726.821336] Lustre: server umount lustre-OST0001 complete [10734.527302] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [10746.983922] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [10794.605417] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [10804.488465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10804.749958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10804.782243] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10804.891366] Lustre: lustre-MDT0000: new disk, initializing [10805.000944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10810.241693] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10820.893581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10821.045209] Lustre: 285216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10821.356079] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10825.258738] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10836.610931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10836.976986] Lustre: 286847:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10838.243293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10843.586757] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10854.372667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10854.589864] Lustre: 287717:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10854.683298] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10854.687920] Lustre: Skipped 6 previous similar messages [10856.610065] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10856.618521] Lustre: Skipped 1 previous similar message [10856.743841] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10861.786630] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10871.813880] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10875.700353] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10875.707508] Lustre: Skipped 1 previous similar message [10891.809700] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 12:07:05 (1788451625) [10896.184748] Lustre: server umount lustre-MDT0000 complete [10897.904682] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10897.916644] LustreError: 288515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10897.931421] Lustre: Skipped 5 previous similar messages [10897.971094] LustreError: 288515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [10899.868450] LustreError: 285206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451634 with bad export cookie 15031984086184719620 [10899.871295] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10899.878279] LustreError: 285206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10900.246546] Lustre: server umount lustre-MDT0001 complete [10914.107249] Lustre: server umount lustre-OST0000 complete [10928.086567] Lustre: server umount lustre-OST0001 complete [10936.731400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10941.495196] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10949.435329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10953.660439] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10963.057807] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10970.035804] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10977.316115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10983.380159] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [10991.808733] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11012.782742] loop1: detected capacity change from 0 to 204800000 [11013.114581] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11013.143731] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11013.173250] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11019.318484] loop1: detected capacity change from 0 to 204800000 [11025.655654] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11025.745637] LustreError: 289955:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs903: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11025.767139] LustreError: 289955:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs903 [11025.775198] LustreError: 289955:0:(mgs_handler.c:517:mgs_target_reg()) fs903-MDT0001: Error checking target index: rc = -13 [11025.790475] LustreError: fs903-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11025.797799] LustreError: 293685:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11025.807392] LustreError: 293685:0:(tgt_mount.c:2129:server_put_super()) no obd fs903-MDT0001 [11025.819984] LustreError: 293685:0:(tgt_mount.c:116:server_deregister_mount()) fs903-MDT0001 not registered [11025.874464] Lustre: server umount fs903-MDT0001 complete [11025.878399] LustreError: 293685:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11034.100995] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11034.178796] Lustre: Setting parameter fs903-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs903-MDT0001 [11034.246632] Lustre: srv-fs903-MDT0001: No data found on store. Initialize space. [11034.257147] Lustre: Skipped 4 previous similar messages [11034.357777] Lustre: fs903-MDT0001: new disk, initializing [11034.362454] Lustre: Skipped 3 previous similar messages [11034.500749] Lustre: srv-fs903-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11034.523283] Lustre: 294151:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs903-MDT0001-mdtlov: can't create new llog: rc = -115 [11034.534710] LustreError: 294151:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs903-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11038.958874] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11045.411532] Lustre: server umount fs903-MDT0001 complete [11059.684938] loop1: detected capacity change from 0 to 204800000 [11061.754564] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11065.981553] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11066.446119] Lustre: 295720:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs903-MDT0001-mdtlov: can't create new llog: rc = -115 [11066.461221] LustreError: 295720:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs903-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11071.202283] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11076.165238] Lustre: server umount fs903-MDT0001 complete [11077.125357] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11087.336042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11087.341221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11087.370049] Lustre: Skipped 1 previous similar message [11087.376556] Lustre: Skipped 2 previous similar messages [11091.915319] Lustre: server umount lustre-MDT0000 complete [11101.282856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11101.451347] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11106.893290] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11107.315836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11115.411803] loop1: detected capacity change from 0 to 204800000 [11121.865225] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11122.242173] Lustre: 298168:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs903-MDT0001-mdtlov: can't create new llog: rc = -115 [11122.259412] LustreError: 298168:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs903-MDT0001-osd: get update log duration 1, retries 0, failed: rc = -115 [11127.187520] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11132.465952] Lustre: server umount fs903-MDT0001 complete [11153.892875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11153.899131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11153.901202] Lustre: Skipped 3 previous similar messages [11158.499417] LustreError: 291797:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11158.534627] LustreError: 291797:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 108 previous similar messages [11159.720967] Lustre: server umount lustre-OST0000 complete [11170.887825] Lustre: server umount lustre-MDT0000 complete [11175.205398] LustreError: 294696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451909 with bad export cookie 15031984086184722301 [11175.209143] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11175.218943] LustreError: 294696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11175.691389] Lustre: server umount lustre-MDT0001 complete [11187.304942] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [11190.525437] Key type lgssc unregistered [11190.979120] LNet: 300516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11190.988097] LNetError: 300516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11191.011953] LNet: Removed LNI 192.168.203.147@tcp [11191.865184] Key type .llcrypt unregistered [11191.866914] Key type ._llcrypt unregistered [11216.456104] Key type ._llcrypt registered [11216.459678] Key type .llcrypt registered [11216.548368] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [11217.861494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11217.885684] alg: No test for adler32 (adler32-zlib) [11219.169742] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11219.558378] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [11221.304180] Key type lgssc registered [11222.322370] Lustre: Echo OBD driver; http://www.lustre.org/ [11230.296795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11230.315286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11231.738835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11236.703879] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11241.957070] LustreError: 301789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11241.966907] LustreError: 301789:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11245.329205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11245.794721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11249.958832] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11258.067803] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11263.537371] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11270.995447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11271.323830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11276.463540] LustreError: 303617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11278.328940] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11281.897747] LustreError: 303616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11287.022157] LustreError: 303617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11288.321757] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11291.102918] LustreError: 303617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11291.152594] LustreError: 303617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11301.359249] LustreError: 303616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11301.391916] LustreError: 303616:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11312.097259] loop1: detected capacity change from 0 to 204800000 [11312.297274] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11312.312171] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11312.334464] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11317.323278] loop1: detected capacity change from 0 to 204800000 [11321.837544] LustreError: 303616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11321.858961] LustreError: 303616:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [11322.552861] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11322.699223] Lustre: fs7889-OST0000: new disk, initializing [11322.701538] Lustre: srv-fs7889-OST0000: No data found on store. Initialize space. [11322.708332] Lustre: 305522:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11322.760447] Lustre: fs7889-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11328.839550] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11340.383079] Lustre: server umount fs7889-OST0000 complete [11354.393760] loop1: detected capacity change from 0 to 204800000 [11357.674814] LustreError: 305473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11357.698179] LustreError: 305473:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [11357.705981] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11360.598056] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11360.943182] Lustre: fs7889-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11368.045899] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11373.054587] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11377.289562] loop2: detected capacity change from 0 to 204800000 [11377.479221] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11377.501960] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11377.526214] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11383.058548] loop2: detected capacity change from 0 to 204800000 [11388.515135] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11388.596951] LustreError: fs7889-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11388.615155] LustreError: fs7889-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11388.622757] LustreError: 308566:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11388.630782] LustreError: 308566:0:(tgt_mount.c:2129:server_put_super()) no obd fs7889-OST0001 [11388.636612] LustreError: 308566:0:(tgt_mount.c:116:server_deregister_mount()) fs7889-OST0001 not registered [11388.658398] Lustre: server umount fs7889-OST0001 complete [11388.661908] LustreError: 308566:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11399.092231] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11399.241544] Lustre: fs7889-OST0001: new disk, initializing [11399.249875] Lustre: srv-fs7889-OST0001: No data found on store. Initialize space. [11399.269382] Lustre: 309117:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11399.353296] Lustre: fs7889-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11406.154706] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11415.674772] Lustre: server umount fs7889-OST0000 complete [11424.235747] LustreError: 303619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11424.261944] LustreError: 303619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [11426.511179] Lustre: server umount fs7889-OST0001 complete [11442.861521] loop1: detected capacity change from 0 to 204800000 [11448.787255] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11448.975272] Lustre: fs7889-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11455.993800] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11463.232996] loop2: detected capacity change from 0 to 204800000 [11469.500858] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11476.435754] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11486.105431] Lustre: server umount fs7889-OST0000 complete [11495.414351] Lustre: server umount fs7889-OST0001 complete [11511.268284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11511.275108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11511.278856] LustreError: Skipped 1 previous similar message [11511.281601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11511.291675] Lustre: Skipped 1 previous similar message [11511.303603] Lustre: Skipped 1 previous similar message [11513.853505] Lustre: server umount lustre-OST0000 complete [11521.512389] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11521.530090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11521.540470] Lustre: Skipped 1 previous similar message [11524.329563] Lustre: server umount lustre-MDT0000 complete [11527.557791] LustreError: 301770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452262 with bad export cookie 18186724754728780883 [11527.566870] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11527.569301] LustreError: 301770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11527.860550] Lustre: server umount lustre-MDT0001 complete [11537.919133] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [11540.725873] Key type lgssc unregistered [11541.085597] LNet: 315043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11541.092319] LNetError: 315043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11541.116686] LNet: Removed LNI 192.168.203.147@tcp [11542.069334] Key type .llcrypt unregistered [11542.073851] Key type ._llcrypt unregistered [11574.652549] Key type ._llcrypt registered [11574.653784] Key type .llcrypt registered [11574.726837] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [11595.538567] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [11596.962876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11597.025767] alg: No test for adler32 (adler32-zlib) [11598.206148] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11598.526597] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [11600.264290] Key type lgssc registered [11601.587147] Lustre: Echo OBD driver; http://www.lustre.org/ [11641.315863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11641.354272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11642.625261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11642.682715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11642.799656] Lustre: lustre-MDT0000: new disk, initializing [11642.895235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11642.919980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11648.208199] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11659.116288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11659.242616] Lustre: 318904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11659.291384] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11659.294490] Lustre: Skipped 1 previous similar message [11659.346879] Lustre: lustre-MDT0001: new disk, initializing [11659.443968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11659.477820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11659.493924] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11664.130939] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11673.562476] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11679.955484] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11687.630196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11687.999914] Lustre: lustre-OST0000: new disk, initializing [11688.004721] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11688.012459] Lustre: 320868:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11688.136551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11695.294828] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11696.247556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11696.269550] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11696.436250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11706.886165] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11709.726661] Lustre: Mounted lustre-client - version 2.17.56_51_g2aa7cea [11711.138906] Lustre: Unmounted lustre-client [11722.209322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11722.212925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11722.221300] LustreError: Skipped 1 previous similar message [11722.263433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11725.306605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11725.329610] Lustre: Skipped 2 previous similar messages [11728.431597] Lustre: server umount lustre-OST0000 complete [11735.535867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11735.557366] Lustre: Skipped 1 previous similar message [11735.561671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11735.567834] Lustre: Skipped 1 previous similar message [11739.399394] Lustre: server umount lustre-MDT0000 complete [11744.518443] LustreError: 318895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452479 with bad export cookie 3055256984821080090 [11744.525711] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11744.539768] LustreError: 318895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11745.036468] Lustre: server umount lustre-MDT0001 complete [11758.415985] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [11761.530596] Key type lgssc unregistered [11767.584151] LNet: 1 local NIs in recovery (showing 1): 192.168.203.147@tcp [11768.033755] LNet: 323036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11768.050745] LNetError: 323036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11838.761270] LNet: Removed LNI 192.168.203.147@tcp [11839.829899] Key type .llcrypt unregistered [11839.834850] Key type ._llcrypt unregistered [11860.896318] Key type ._llcrypt registered [11860.900211] Key type .llcrypt registered [11861.051475] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [11878.322612] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [11879.832658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11879.915643] alg: No test for adler32 (adler32-zlib) [11881.061814] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11881.279158] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [11882.920170] Key type lgssc registered [11884.127318] Lustre: Echo OBD driver; http://www.lustre.org/ [11914.652372] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 12:24:07 (1788452647) [11921.373445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11921.410378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11922.652235] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11922.687914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11922.774822] Lustre: lustre-MDT0000: new disk, initializing [11922.872367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11922.893057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11928.424337] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11939.499828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11939.632656] Lustre: 326262:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11939.712621] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11939.717958] Lustre: Skipped 1 previous similar message [11939.842605] Lustre: lustre-MDT0001: new disk, initializing [11939.956104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11939.999116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11940.026208] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11944.592225] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11953.856940] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11959.901405] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11966.282996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11966.607750] Lustre: lustre-OST0000: new disk, initializing [11966.617951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11966.635269] Lustre: 328230:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11966.757168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11972.744623] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [11976.257852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11976.276575] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11976.419385] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11981.950216] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11997.157902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11997.175419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11997.175952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11997.196907] Lustre: Skipped 1 previous similar message [12002.282434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12002.288855] Lustre: Skipped 1 previous similar message [12002.468835] Lustre: server umount lustre-OST0000 complete [12006.375508] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12006.379403] LustreError: Skipped 1 previous similar message [12006.383727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12006.394228] Lustre: Skipped 1 previous similar message [12006.397979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12007.396290] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12012.516651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12012.530320] Lustre: Skipped 3 previous similar messages [12012.662268] Lustre: server umount lustre-MDT0000 complete [12016.429823] LustreError: 326253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452751 with bad export cookie 1267760609843173555 [12016.435174] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12016.443084] LustreError: 326253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12016.835643] Lustre: server umount lustre-MDT0001 complete [12028.332188] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [12031.768466] Key type lgssc unregistered [12032.077955] LNet: 330139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12032.086042] LNetError: 330139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12032.106967] LNet: Removed LNI 192.168.203.147@tcp [12033.116198] Key type .llcrypt unregistered [12033.119127] Key type ._llcrypt unregistered [12058.256472] Key type ._llcrypt registered [12058.258686] Key type .llcrypt registered [12058.354490] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [12059.883360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12059.912179] alg: No test for adler32 (adler32-zlib) [12061.081604] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12061.393647] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [12063.096161] Key type lgssc registered [12064.368935] Lustre: Echo OBD driver; http://www.lustre.org/ [12071.660829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12071.680475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12073.134753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12077.789827] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12082.661105] LustreError: 331413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12082.682478] LustreError: 331413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12086.149588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12086.648950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12091.512526] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12099.838565] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12106.211437] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12114.532905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12115.105066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12121.600907] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12129.395986] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12140.536860] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12142.283336] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12147.172467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12147.184628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12147.195422] Lustre: Skipped 2 previous similar messages [12147.230263] Lustre: Skipped 2 previous similar messages [12147.840027] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.47@tcp (stopping) [12152.293277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12152.301196] Lustre: Skipped 2 previous similar messages [12153.189525] Lustre: server umount lustre-MDT0000 complete [12157.028737] LustreError: 331393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452891 with bad export cookie 13976525000880879941 [12157.033393] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12157.039138] LustreError: 331393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12157.414461] LustreError: 332418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12157.415130] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12157.464983] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12157.473440] Lustre: Skipped 1 previous similar message [12162.530099] LustreError: 333254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12162.530351] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12162.540733] LustreError: 333254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12162.557574] Lustre: Skipped 1 previous similar message [12163.410650] Lustre: server umount lustre-MDT0001 complete [12168.059587] Lustre: server umount lustre-OST0000 complete [12179.088635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12185.885709] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12188.474890] Lustre: 335595:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.147@tcp [12190.713649] Lustre: server umount MGS complete [12199.806730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12200.491531] LustreError: 336168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12200.519179] LustreError: 336168:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12200.631135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12205.685929] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12206.048868] LustreError: 336169:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12215.240612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12215.874894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12220.374515] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12230.870855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12231.147715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12237.661772] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12245.330440] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12247.097625] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12252.820972] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12254.552993] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12260.833763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12260.834679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12260.849433] LustreError: Skipped 1 previous similar message [12260.852476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12260.876644] Lustre: Skipped 1 previous similar message [12260.899836] Lustre: Skipped 2 previous similar messages [12261.561457] Lustre: server umount lustre-OST0000 complete [12265.954645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12271.717966] Lustre: server umount lustre-MDT0000 complete [12275.684691] LustreError: 337626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453010 with bad export cookie 13976525000880880781 [12275.690623] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12275.696379] LustreError: 337626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12276.093158] Lustre: server umount lustre-MDT0001 complete [12288.532755] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [12292.283926] Key type lgssc unregistered [12292.722324] LNet: 339607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12292.736188] LNetError: 339607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12292.777625] LNet: Removed LNI 192.168.203.147@tcp [12293.953383] Key type .llcrypt unregistered [12293.956328] Key type ._llcrypt unregistered [12317.037467] Key type ._llcrypt registered [12317.041067] Key type .llcrypt registered [12317.151304] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [12335.827346] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [12337.196298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12337.331079] alg: No test for adler32 (adler32-zlib) [12338.406815] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12338.779626] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [12340.561410] Key type lgssc registered [12341.832792] Lustre: Echo OBD driver; http://www.lustre.org/ [12387.166264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12387.201376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12388.516855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12388.619778] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12388.731105] Lustre: lustre-MDT0000: new disk, initializing [12388.816910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12388.844097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12394.157236] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12405.548475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12405.708646] Lustre: 342769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12405.781653] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12405.787618] Lustre: Skipped 1 previous similar message [12405.880381] Lustre: lustre-MDT0001: new disk, initializing [12405.985143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12406.021648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12406.034442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12411.199425] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12421.765465] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12428.148570] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12435.628870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12436.004853] Lustre: lustre-OST0000: new disk, initializing [12436.014509] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12436.020663] Lustre: 344741:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12436.131205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12442.700821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12442.721257] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12442.879251] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12443.435873] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12454.156288] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12460.109884] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12460.356909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12466.485062] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12466.864125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12473.825790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12473.835523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12473.839912] LustreError: Skipped 1 previous similar message [12473.843460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12473.847627] Lustre: Skipped 1 previous similar message [12473.866485] Lustre: Skipped 1 previous similar message [12475.498961] Lustre: server umount lustre-OST0000 complete [12483.041903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12483.055436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12483.067515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12484.066042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12484.092409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12484.109475] Lustre: Skipped 1 previous similar message [12486.004752] Lustre: server umount lustre-MDT0000 complete [12489.990655] LustreError: 342764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453224 with bad export cookie 15326323860833779747 [12489.998869] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12489.999171] LustreError: 342764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12490.325119] Lustre: server umount lustre-MDT0001 complete [12501.037980] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 12:33:53 (1788453233) [12519.146058] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [12530.463404] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [12570.129578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12570.403449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12570.482942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12570.581856] Lustre: lustre-MDT0000: new disk, initializing [12570.676377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12570.701120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12575.729584] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12586.491878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12586.684337] Lustre: 349500:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12586.734579] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12586.738742] Lustre: Skipped 1 previous similar message [12586.781297] Lustre: lustre-MDT0001: new disk, initializing [12586.884203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12586.913591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12586.923987] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12591.260426] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12602.553920] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12610.486369] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12620.061740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12620.384785] Lustre: lustre-OST0000: new disk, initializing [12620.390859] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12620.401314] Lustre: 351470:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12620.486243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12627.461490] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12629.049686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12629.062593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12629.160536] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12637.428086] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12680.678298] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12680.689541] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12680.692209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12680.702601] Lustre: Skipped 1 previous similar message [12680.726557] Lustre: Skipped 1 previous similar message [12685.314575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12685.336082] Lustre: Skipped 1 previous similar message [12686.901602] Lustre: server umount lustre-OST0000 complete [12694.497975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12694.508642] LustreError: Skipped 1 previous similar message [12694.511984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12694.564186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12697.486840] Lustre: server umount lustre-MDT0000 complete [12700.641172] LustreError: 351220:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12700.669838] LustreError: 351220:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12701.783747] LustreError: 350307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453436 with bad export cookie 15326323860833780622 [12701.790937] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12701.802024] LustreError: 350307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12702.371673] Lustre: server umount lustre-MDT0001 complete [12715.808510] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [12718.793296] Key type lgssc unregistered [12719.063048] LNet: 353649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12719.078311] LNetError: 353649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12719.105256] LNet: Removed LNI 192.168.203.147@tcp [12719.754727] Key type .llcrypt unregistered [12719.758141] Key type ._llcrypt unregistered [12741.018193] Key type ._llcrypt registered [12741.025198] Key type .llcrypt registered [12741.179536] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [12742.572665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12742.705344] alg: No test for adler32 (adler32-zlib) [12743.941765] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12744.160087] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [12745.880163] Key type lgssc registered [12747.378838] Lustre: Echo OBD driver; http://www.lustre.org/ [12755.888994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12755.943657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12757.413307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12762.220589] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12767.722654] LustreError: 354925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12767.752946] LustreError: 354925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12770.302489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12770.610252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12776.565199] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12786.653861] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12793.741072] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12800.732282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12801.225535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12808.161489] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12817.019583] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12829.158993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12829.183039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12829.227082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12831.207201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12831.208974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12831.209297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12831.281541] Lustre: Skipped 1 previous similar message [12834.913421] Lustre: server umount lustre-OST0000 complete [12841.446786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12841.452743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12841.461768] Lustre: Skipped 1 previous similar message [12841.482587] Lustre: Skipped 1 previous similar message [12845.982380] Lustre: server umount lustre-MDT0000 complete [12846.567254] LustreError: 354924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12846.589626] LustreError: 354924:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12850.396291] LustreError: 354907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453585 with bad export cookie 17940994117606147850 [12850.414452] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12851.167285] Lustre: server umount lustre-MDT0001 complete [12863.464611] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [12867.184852] Key type lgssc unregistered [12867.623505] LNet: 360390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12867.632514] LNetError: 360390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12867.674262] LNet: Removed LNI 192.168.203.147@tcp [12868.743104] Key type .llcrypt unregistered [12868.752887] Key type ._llcrypt unregistered [12887.264483] Key type ._llcrypt registered [12887.266483] Key type .llcrypt registered [12887.387932] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [12888.649430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12888.816333] alg: No test for adler32 (adler32-zlib) [12890.116276] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12890.471152] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [12892.223056] Key type lgssc registered [12893.629151] Lustre: Echo OBD driver; http://www.lustre.org/ [12902.858068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12902.874911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12904.465891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12910.766259] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12914.665742] LustreError: 364127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12914.708915] LustreError: 364127:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12919.782877] LustreError: 364126:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12921.943594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12922.387241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12928.216956] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12938.128092] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12945.094584] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12953.953184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12954.452340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12961.760078] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [12969.841746] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12983.277177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12983.287739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12983.313801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12985.326411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12985.331364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12985.363997] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12988.589913] Lustre: server umount lustre-OST0000 complete [12994.026381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12994.041536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12994.073376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12994.086126] Lustre: Skipped 1 previous similar message [13000.679209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13000.685404] Lustre: Skipped 3 previous similar messages [13005.797701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13005.809887] Lustre: Skipped 1 previous similar message [13007.328080] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13007.747248] Lustre: server umount lustre-MDT0000 complete [13010.924454] LustreError: 364127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13010.957990] LustreError: 364127:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13012.052074] LustreError: 365490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453746 with bad export cookie 4008821578836943235 [13012.073238] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13012.071995] LustreError: 365490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13012.552895] Lustre: server umount lustre-MDT0001 complete [13025.267776] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [13029.054696] Key type lgssc unregistered [13029.561181] LNet: 371981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13029.567324] LNetError: 371981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13029.588748] LNet: Removed LNI 192.168.203.147@tcp [13030.971630] Key type .llcrypt unregistered [13030.978290] Key type ._llcrypt unregistered [13050.759155] Key type ._llcrypt registered [13050.761277] Key type .llcrypt registered [13050.994037] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [13052.414289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13052.523777] alg: No test for adler32 (adler32-zlib) [13053.700869] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13054.051733] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [13055.848199] Key type lgssc registered [13057.639367] Lustre: Echo OBD driver; http://www.lustre.org/ [13066.581501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13066.635408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13068.241991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13072.879369] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [13077.476740] LustreError: 375741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13077.494653] LustreError: 375741:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13082.606508] LustreError: 375742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13082.663444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13083.283347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13088.808061] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [13098.228292] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13106.353504] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13116.254592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13116.907402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13126.367767] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [13136.418902] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13152.737674] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13152.742107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13152.756667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13152.768587] Lustre: Skipped 1 previous similar message [13157.486106] Lustre: server umount lustre-OST0000 complete [13162.980115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13162.987502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13163.009146] Lustre: Skipped 2 previous similar messages [13163.013672] Lustre: Skipped 1 previous similar message [13168.097620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13168.105347] Lustre: Skipped 1 previous similar message [13173.221037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13173.238863] Lustre: Skipped 1 previous similar message [13175.264324] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13175.488973] Lustre: server umount lustre-MDT0000 complete [13178.342875] LustreError: 375741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13178.363540] LustreError: 375741:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13180.677415] LustreError: 375672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453915 with bad export cookie 6293356104761510679 [13180.685555] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13180.687374] LustreError: 375672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13180.910649] Lustre: server umount lustre-MDT0001 complete [13194.334636] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [13196.911245] Key type lgssc unregistered [13197.180727] LNet: 381845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13197.187883] LNetError: 381845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13197.210263] LNet: Removed LNI 192.168.203.147@tcp [13197.979050] Key type .llcrypt unregistered [13197.981476] Key type ._llcrypt unregistered [13297.608578] Key type ._llcrypt registered [13297.614026] Key type .llcrypt registered [13297.738290] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [13298.039287] Key type .llcrypt unregistered [13298.041082] Key type ._llcrypt unregistered [13385.328211] Key type ._llcrypt registered [13385.330629] Key type .llcrypt registered [13385.410931] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [13385.665310] Key type .llcrypt unregistered [13385.670863] Key type ._llcrypt unregistered [13419.487235] Key type ._llcrypt registered [13419.488869] Key type .llcrypt registered [13419.634916] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [13420.032722] Key type .llcrypt unregistered [13420.035679] Key type ._llcrypt unregistered [13457.165513] Key type ._llcrypt registered [13457.166834] Key type .llcrypt registered [13457.269752] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [13457.575938] Key type .llcrypt unregistered [13457.582186] Key type ._llcrypt unregistered [13534.216762] Key type ._llcrypt registered [13534.225138] Key type .llcrypt registered [13534.330436] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [13534.624451] Key type .llcrypt unregistered [13534.627617] Key type ._llcrypt unregistered [13567.927712] Key type ._llcrypt registered [13567.930339] Key type .llcrypt registered [13568.185959] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [13568.659779] Key type .llcrypt unregistered [13568.666593] Key type ._llcrypt unregistered [13602.282904] Key type ._llcrypt registered [13602.285872] Key type .llcrypt registered [13602.449345] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [13602.876946] Key type .llcrypt unregistered [13602.880021] Key type ._llcrypt unregistered [13633.630541] Key type ._llcrypt registered [13633.632661] Key type .llcrypt registered [13633.783266] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [13634.188067] Key type .llcrypt unregistered [13634.189870] Key type ._llcrypt unregistered [13664.948533] Key type ._llcrypt registered [13664.951253] Key type .llcrypt registered [13665.111581] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [13665.539659] Key type .llcrypt unregistered [13665.542508] Key type ._llcrypt unregistered [13698.931984] Key type ._llcrypt registered [13698.935300] Key type .llcrypt registered [13699.002446] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [13699.224321] Key type .llcrypt unregistered [13699.228705] Key type ._llcrypt unregistered [13765.146880] Key type ._llcrypt registered [13765.151047] Key type .llcrypt registered [13765.221770] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [13765.529279] Key type .llcrypt unregistered [13765.535114] Key type ._llcrypt unregistered [13797.385335] Key type ._llcrypt registered [13797.389672] Key type .llcrypt registered [13797.479661] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [13797.857504] Key type .llcrypt unregistered [13797.862331] Key type ._llcrypt unregistered [13828.419716] Key type ._llcrypt registered [13828.424871] Key type .llcrypt registered [13828.516818] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [13828.764302] Key type .llcrypt unregistered [13828.766455] Key type ._llcrypt unregistered [13887.949428] Key type ._llcrypt registered [13887.953185] Key type .llcrypt registered [13888.098746] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [13904.064135] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [13905.826899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13905.876637] alg: No test for adler32 (adler32-zlib) [13907.039735] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13907.365857] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [13909.168249] Key type lgssc registered [13910.808602] Lustre: Echo OBD driver; http://www.lustre.org/ [13950.234925] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13950.286531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13951.530740] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13951.564745] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13951.627433] Lustre: tst250fs-MDT0000: new disk, initializing [13951.672420] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13951.688193] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13955.903707] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [13970.245158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13970.515797] Lustre: 402486:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13970.627780] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13970.631515] Lustre: Skipped 1 previous similar message [13970.705230] Lustre: tst250fs-MDT0001: new disk, initializing [13970.828513] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13970.860872] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13970.869878] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13978.257968] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [13984.482007] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13992.843502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13993.073312] Lustre: tst250fs-OST0000: new disk, initializing [13993.076747] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13993.080415] Lustre: 404121:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13993.143395] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13999.653302] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13999.681874] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13999.783831] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14000.652729] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [14013.855995] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14014.176737] Lustre: tst250fs-OST0001: new disk, initializing [14014.185050] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14014.196275] Lustre: 404996:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14014.341894] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14019.727496] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14019.743225] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14019.895489] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14022.332481] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [14028.988506] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14031.059413] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14031.071835] Lustre: Skipped 1 previous similar message [14055.911134] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14055.916043] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14055.931343] Lustre: Skipped 3 previous similar messages [14055.937629] Lustre: Skipped 2 previous similar messages [14059.229719] Lustre: server umount tst250fs-MDT0000 complete [14064.234977] LustreError: 402481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454798 with bad export cookie 8283423064836699330 [14064.243543] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14064.251632] LustreError: 402481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14065.116821] Lustre: server umount tst250fs-MDT0001 complete [14080.952764] Lustre: server umount tst250fs-OST0000 complete [14081.318706] Lustre: 400712:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454800/real 1788454800] req@ffff9215c33d8000 x1875330614765056/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454816 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14081.355821] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14084.321227] Lustre: 400710:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454803/real 1788454803] req@ffff9215cc8fca80 x1875330614765312/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454819 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14085.664468] Lustre: server umount tst250fs-OST0001 complete [14106.795490] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [14116.613859] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [14159.832800] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 13:01:32 (1788454892) [14162.890576] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14183.184555] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [14196.307584] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [14235.096384] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13982 sec ========= 13:02:47 (1788454967) [14237.820645] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:02:50 (1788454970) === [14241.442471] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:02:54 (1788454974) === [14292.245217] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [14296.280380] Key type lgssc unregistered [14296.550640] LNet: 414154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14296.566551] LNetError: 414154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14296.601866] LNet: Removed LNI 192.168.203.147@tcp [14297.675223] Key type .llcrypt unregistered [14297.683329] Key type ._llcrypt unregistered