[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 550009601 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003078] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008482] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009055] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010022] pid_max: default: 32768 minimum: 301 [ 0.011172] LSM: Security Framework initializing [ 0.012082] Yama: becoming mindful. [ 0.013055] SELinux: Initializing. [ 0.014083] *** VALIDATE selinux *** [ 0.022820] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027292] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028195] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030131] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031169] *** VALIDATE tmpfs *** [ 0.033278] *** VALIDATE proc *** [ 0.034336] *** VALIDATE cgroup *** [ 0.035016] *** VALIDATE cgroup2 *** [ 0.037035] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038183] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039020] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040043] Spectre V2 : User space: Vulnerable [ 0.041016] Speculative Store Bypass: Vulnerable [ 0.044236] debug: unmapping init [mem 0xffffffff94c59000-0xffffffff94c60fff] [ 0.047212] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048809] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049035] ... version: 2 [ 0.050020] ... bit width: 48 [ 0.051019] ... generic registers: 4 [ 0.052023] ... value mask: 0000ffffffffffff [ 0.053020] ... max period: 00007fffffffffff [ 0.054022] ... fixed-purpose events: 3 [ 0.055021] ... event mask: 000000070000000f [ 0.056409] rcu: Hierarchical SRCU implementation. [ 0.058893] smp: Bringing up secondary CPUs ... [ 0.059730] x86: Booting SMP configuration: [ 0.060039] .... node #0, CPUs: #1 #2 #3 [ 0.064756] smp: Brought up 1 node, 4 CPUs [ 0.066025] smpboot: Max logical packages: 1 [ 0.067018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.240548] node 0 deferred pages initialised in 170ms [ 0.242257] devtmpfs: initialized [ 0.243284] x86/mm: Memory block size: 128MB [ 0.245892] gcov: version magic: 0x41383552 [ 0.247464] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.250108] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.253468] pinctrl core: initialized pinctrl subsystem [ 0.255231] [ 0.255792] ************************************************************* [ 0.258021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.260018] ** ** [ 0.262018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.265020] ** ** [ 0.267017] ** This means that this kernel is built to expose internal ** [ 0.269019] ** IOMMU data structures, which may compromise security on ** [ 0.272015] ** your system. ** [ 0.274015] ** ** [ 0.276016] ** If you see this message and you are not debugging the ** [ 0.278029] ** kernel, report this immediately to your vendor! ** [ 0.280015] ** ** [ 0.283018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.285015] ************************************************************* [ 0.287788] NET: Registered protocol family 16 [ 0.289538] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.292071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.294092] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.297514] cpuidle: using governor menu [ 0.298602] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.301627] PCI: Using configuration type 1 for base access [ 0.304140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.313194] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.315040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.318098] cryptd: max_cpu_qlen set to 1000 [ 0.319237] ACPI: Added _OSI(Module Device) [ 0.320009] ACPI: Added _OSI(Processor Device) [ 0.321015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.323022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.327984] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.332483] ACPI: Interpreter enabled [ 0.333065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.334011] ACPI: Using IOAPIC for interrupt routing [ 0.335080] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.338584] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.348251] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.350045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.352022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.355114] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.359359] acpiphp: Slot [2] registered [ 0.360098] acpiphp: Slot [5] registered [ 0.361084] acpiphp: Slot [6] registered [ 0.362079] acpiphp: Slot [7] registered [ 0.362918] acpiphp: Slot [8] registered [ 0.364087] acpiphp: Slot [9] registered [ 0.365033] acpiphp: Slot [10] registered [ 0.366125] acpiphp: Slot [3] registered [ 0.368077] acpiphp: Slot [4] registered [ 0.368923] acpiphp: Slot [11] registered [ 0.369055] acpiphp: Slot [12] registered [ 0.369819] acpiphp: Slot [13] registered [ 0.371070] acpiphp: Slot [14] registered [ 0.371911] acpiphp: Slot [15] registered [ 0.372052] acpiphp: Slot [16] registered [ 0.372858] acpiphp: Slot [17] registered [ 0.373085] acpiphp: Slot [18] registered [ 0.374000] acpiphp: Slot [19] registered [ 0.374828] acpiphp: Slot [20] registered [ 0.376055] acpiphp: Slot [21] registered [ 0.376833] acpiphp: Slot [22] registered [ 0.377070] acpiphp: Slot [23] registered [ 0.377985] acpiphp: Slot [24] registered [ 0.379079] acpiphp: Slot [25] registered [ 0.379870] acpiphp: Slot [26] registered [ 0.381064] acpiphp: Slot [27] registered [ 0.381780] acpiphp: Slot [28] registered [ 0.382054] acpiphp: Slot [29] registered [ 0.382955] acpiphp: Slot [30] registered [ 0.384075] acpiphp: Slot [31] registered [ 0.385065] PCI host bridge to bus 0000:00 [ 0.387020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.389028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.391028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.393029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.395024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.397026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.399175] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.400782] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.402856] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.411029] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.416030] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.417014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.419012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.420011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.422505] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.423635] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.425027] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.427542] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.430011] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.438931] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.444017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.447707] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.452015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.457023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.473016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.480550] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.487025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.492013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.512033] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.524414] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.530021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.538023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.555020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.564607] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.572020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.577020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.592026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.600665] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.609018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.617023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.641019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.654270] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.659016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.668021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.685025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.695510] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.698423] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.701373] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.703459] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.706256] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.713050] iommu: Default domain type: Passthrough [ 0.714534] SCSI subsystem initialized [ 0.716170] ACPI: bus type USB registered [ 0.718133] usbcore: registered new interface driver usbfs [ 0.719139] usbcore: registered new interface driver hub [ 0.721086] usbcore: registered new device driver usb [ 0.722241] pps_core: LinuxPPS API ver. 1 registered [ 0.724022] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.727074] PTP clock support registered [ 0.729157] EDAC MC: Ver: 3.0.0 [ 0.732147] PCI: Using ACPI for IRQ routing [ 0.733951] NetLabel: Initializing [ 0.735011] NetLabel: domain hash size = 128 [ 0.737014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.739090] NetLabel: unlabeled traffic allowed by default [ 0.741162] vgaarb: loaded [ 0.742299] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.744019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.750339] clocksource: Switched to clocksource kvm-clock [ 0.859803] VFS: Disk quotas dquot_6.6.0 [ 0.861485] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.863731] *** VALIDATE ramfs *** [ 0.865163] *** VALIDATE hugetlbfs *** [ 0.867533] pnp: PnP ACPI init [ 0.870057] pnp: PnP ACPI: found 6 devices [ 0.887814] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.891304] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.893521] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.895720] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.897671] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.899803] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.902834] NET: Registered protocol family 2 [ 0.905468] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.910408] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.914292] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.920745] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.924793] TCP: Hash tables configured (established 65536 bind 65536) [ 0.928228] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.931649] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.934947] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.938599] NET: Registered protocol family 1 [ 0.941536] RPC: Registered named UNIX socket transport module. [ 0.944345] RPC: Registered udp transport module. [ 0.946386] RPC: Registered tcp transport module. [ 0.947652] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.949511] NET: Registered protocol family 44 [ 0.951692] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.954166] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.955980] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.958331] PCI: CLS 0 bytes, default 64 [ 0.959910] Unpacking initramfs... [ 2.374277] debug: unmapping init [mem 0xffff9b57bcc54000-0xffff9b57bffbffff] [ 2.378412] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.380420] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.383037] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.889443] Initialise system trusted keyrings [ 2.891156] Key type blacklist registered [ 2.893424] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.903082] zbud: loaded [ 2.906145] *** VALIDATE nfs *** [ 2.907401] *** VALIDATE nfs4 *** [ 2.909035] pstore: using deflate compression [ 2.912306] Platform Keyring initialized [ 3.029041] NET: Registered protocol family 38 [ 3.030713] Key type asymmetric registered [ 3.032113] Asymmetric key parser 'x509' registered [ 3.034155] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.038070] io scheduler mq-deadline registered [ 3.040283] io scheduler kyber registered [ 3.042903] io scheduler bfq registered [ 3.045839] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.049540] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.052807] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.055490] ACPI: Power Button [PWRF] [ 3.061529] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.068535] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.085732] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.099724] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.124680] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.152908] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.182672] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.187962] Non-volatile memory driver v1.3 [ 3.189701] Linux agpgart interface v0.103 [ 3.224487] virtio_blk virtio1: [vda] 146640 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.227409] vda: detected capacity change from 0 to 75079680 [ 3.250092] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.253373] vdb: detected capacity change from 0 to 1073741824 [ 3.269890] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.272946] vdc: detected capacity change from 0 to 2621440000 [ 3.290923] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.293992] vdd: detected capacity change from 0 to 2621440000 [ 3.311163] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.314135] vde: detected capacity change from 0 to 4294967296 [ 3.331081] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.334410] vdf: detected capacity change from 0 to 4294967296 [ 3.345959] libphy: Fixed MDIO Bus: probed [ 3.351553] usbcore: registered new interface driver usbserial_generic [ 3.354070] usbserial: USB Serial support registered for generic [ 3.356929] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.362440] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.364353] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.366880] mousedev: PS/2 mouse device common for all mice [ 3.370560] rtc_cmos 00:05: RTC can wake from S4 [ 3.370616] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.373602] rtc_cmos 00:05: registered as rtc0 [ 3.377701] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.380749] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.381107] intel_pstate: CPU model not supported [ 3.389830] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.390791] hid: raw HID events driver (C) Jiri Kosina [ 3.396586] usbcore: registered new interface driver usbhid [ 3.397849] usbhid: USB HID core driver [ 3.398814] drop_monitor: Initializing network drop monitor service [ 3.400377] Initializing XFRM netlink socket [ 3.401619] NET: Registered protocol family 10 [ 3.403666] Segment Routing with IPv6 [ 3.404669] NET: Registered protocol family 17 [ 3.405917] mpls_gso: MPLS GSO support [ 3.410207] RAS: Correctable Errors collector initialized. [ 3.412609] AVX version of gcm_enc/dec engaged. [ 3.413876] AES CTR mode by8 optimization enabled [ 3.491218] sched_clock: Marking stable (3491189329, 0)->(4501752785, -1010563456) [ 3.493652] registered taskstats version 1 [ 3.495452] Loading compiled-in X.509 certificates [ 3.497420] zswap: loaded using pool lzo/zbud [ 3.525799] Key type big_key registered [ 3.537852] Key type encrypted registered [ 3.539488] ima: No TPM chip found, activating TPM-bypass! [ 3.541481] ima: Allocated hash algorithm: sha1 [ 3.543135] ima: No architecture policies found [ 3.545011] evm: Initialising EVM extended attributes: [ 3.546821] evm: security.selinux [ 3.547960] evm: security.ima [ 3.549019] evm: security.capability [ 3.550270] evm: HMAC attrs: 0x1 [ 3.552650] rtc_cmos 00:05: setting system clock to 2026-09-05 03:52:06 UTC (1788580326) [ 3.558392] debug: unmapping init [mem 0xffffffff95c03000-0xffffffff95dfffff] [ 3.561563] debug: unmapping init [mem 0xffffffff94982000-0xffffffff94c58fff] [ 3.570134] Write protecting the kernel read-only data: 28672k [ 3.573331] debug: unmapping init [mem 0xffffffff93003000-0xffffffff931fffff] [ 3.576019] debug: unmapping init [mem 0xffffffff93914000-0xffffffff939fffff] [ 3.610246] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.618362] systemd[1]: Detected virtualization kvm. [ 3.620377] systemd[1]: Detected architecture x86-64. [ 3.622069] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.645669] systemd[1]: No hostname configured. [ 3.647491] systemd[1]: Set hostname to . [ 3.649483] random: systemd: uninitialized urandom read (16 bytes read) [ 3.652104] systemd[1]: Initializing machine ID from random generator. [ 3.682472] random: ln: uninitialized urandom read (6 bytes read) [ 3.792482] random: systemd: uninitialized urandom read (16 bytes read) [ 3.796059] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.801466] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.805992] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.444797] device-mapper: uevent: version 1.0.3 [ 4.446435] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.293406] virtio_net virtio0 ens2: renamed from eth0 [ 5.296733] random: fast init done [ 5.391488] scsi host0: ata_piix [ 5.415897] scsi host1: ata_piix [ 5.417610] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.420093] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.096698] random: crng init done [ 10.106409] random: 7 urandom warning(s) missed due to ratelimiting [ 10.504033] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 13.030122] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.400743] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.456929] SELinux: Disabled at runtime. [ 17.608434] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.640844] systemd[1]: Detected virtualization kvm. [ 17.648091] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.828650] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.835219] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.847720] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.855601] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.861091] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.875373] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.887545] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. [ 20.550934] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 21.455291] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.720887] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.874629] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.391964] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.459814] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 28.879804] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 29.488135] NFS: Registering the id_resolver key type [ 29.490350] Key type id_resolver registered [ 29.492403] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg207-server login: [ 80.894015] hrtimer: interrupt took 2219593 ns [ 98.776327] libcfs: loading out-of-tree module taints kernel. [ 98.839626] Key type ._llcrypt registered [ 98.842067] Key type .llcrypt registered [ 98.946223] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 120.607616] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 122.030248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 122.045195] alg: No test for adler32 (adler32-zlib) [ 123.324762] Lustre: Lustre: Build Version: 2.17.57_103_g3d8212a [ 124.196342] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 125.887163] Key type lgssc registered [ 127.848026] Lustre: Echo OBD driver; http://www.lustre.org/ [ 148.287422] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 196.248310] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 210.480467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 210.533444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 211.810683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 211.831745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 211.903671] Lustre: lustre-MDT0000: new disk, initializing [ 211.958305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 211.970305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 216.738917] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.789644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 229.936853] Lustre: 6519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 229.962326] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 229.968285] Lustre: Skipped 1 previous similar message [ 230.070538] Lustre: lustre-MDT0001: new disk, initializing [ 230.129226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 230.168077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 230.185456] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 234.331367] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.747084] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 247.847519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 248.241567] Lustre: lustre-OST0000: new disk, initializing [ 248.253645] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 248.266180] Lustre: 8456:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 248.376760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 251.871635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 251.883359] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 251.975210] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 254.450503] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 269.967704] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 270.067055] Lustre: lustre-OST0001: new disk, initializing [ 270.070596] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 270.074977] Lustre: 9531:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 270.126849] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 276.130186] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 278.572225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 278.585511] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 278.652771] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 290.095942] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 301.856846] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 308.148516] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing check_logdir /tmp/testlogs/ [ 313.668049] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing yml_node [ 318.181676] Lustre: DEBUG MARKER: Client: 2.17.57.103 [ 320.882115] Lustre: DEBUG MARKER: MDS: 2.17.57.103 [ 323.726349] Lustre: DEBUG MARKER: OSS: 2.17.57.103 [ 325.510628] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 4 23:57:26 EDT 2026 [ 343.004390] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 344.778344] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 363.488195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 363.503407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 363.526305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 365.541194] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.554286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 365.557358] Lustre: Skipped 2 previous similar messages [ 365.602226] Lustre: Skipped 3 previous similar messages [ 370.661637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 370.671565] Lustre: Skipped 3 previous similar messages [ 375.775978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 375.790729] Lustre: Skipped 3 previous similar messages [ 376.287151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 376.663456] Lustre: server umount lustre-MDT0000 complete [ 380.758785] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580703 with bad export cookie 9429784141738425337 [ 380.790087] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 380.804704] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 380.895918] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 380.897095] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 380.908375] Lustre: Skipped 1 previous similar message [ 386.017858] LustreError: 6527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 386.041717] LustreError: 6527:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 390.113796] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 390.114598] LustreError: 8921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 390.126804] Lustre: Skipped 4 previous similar messages [ 390.158140] LustreError: 8921:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 395.235254] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 395.235437] LustreError: 6525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 395.266225] LustreError: 6525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 395.451396] Lustre: server umount lustre-MDT0001 complete [ 399.838625] Lustre: server umount lustre-OST0000 complete [ 403.653212] Lustre: server umount lustre-OST0001 complete [ 411.292313] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 419.912958] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 455.963821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 456.152919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 456.177568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 456.253905] Lustre: lustre-MDT0000: new disk, initializing [ 456.307454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 456.321860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 460.282848] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 471.462522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 471.575677] Lustre: 15086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 471.644277] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 471.648639] Lustre: Skipped 1 previous similar message [ 471.757737] Lustre: lustre-MDT0001: new disk, initializing [ 471.832772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 471.860506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 471.878900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 476.004268] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 486.418360] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 492.964372] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 499.676783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 499.896720] Lustre: lustre-OST0000: new disk, initializing [ 499.899426] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 499.904496] Lustre: 17057:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 499.978666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 501.582373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 501.603864] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 501.747427] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 506.426425] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 516.721199] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 521.791471] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 522.044475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 526.514448] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 526.762412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 532.450809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 532.461688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 532.469488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 532.479921] Lustre: Skipped 1 previous similar message [ 532.495869] Lustre: Skipped 3 previous similar messages [ 543.199160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 543.432657] Lustre: server umount lustre-OST0000 complete [ 547.821766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.897761] Lustre: server umount lustre-MDT0000 complete [ 558.050572] LustreError: 15098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.058012] LustreError: 15098:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 558.258964] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580881 with bad export cookie 9429784141738426877 [ 558.268736] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 558.271123] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 558.924657] Lustre: server umount lustre-MDT0001 complete [ 566.011531] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 00:01:26 (1788580886) [ 586.126503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 586.478737] LustreError: 19947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.555759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 591.527534] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 596.967707] LustreError: 19948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 597.003044] LustreError: 19948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 602.375079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 608.507892] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 617.545228] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 624.132698] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 626.635319] Lustre: Failing over lustre-MDT0000 [ 626.927121] Lustre: server umount lustre-MDT0000 complete [ 627.695743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 627.717031] Lustre: Skipped 1 previous similar message [ 627.730139] LustreError: 19948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 627.759503] LustreError: 19948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 643.954886] Lustre: 3650:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788580950/real 1788580950] req@ffff9b5703c44380 x1875462530961024/t0(0) o400->MGC192.168.202.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788580966 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 644.003938] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 646.457711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 653.590272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 653.603419] Lustre: Skipped 1 previous similar message [ 653.675395] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 658.926259] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 658.934356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 659.018648] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 659.274467] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 669.065753] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 674.785786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 674.790518] LustreError: Skipped 1 previous similar message [ 674.801062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 674.816865] Lustre: Skipped 1 previous similar message [ 674.835425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 674.839810] Lustre: Skipped 8 previous similar messages [ 679.559452] Lustre: server umount lustre-MDT0000 complete [ 682.981857] LustreError: 19948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 683.008756] LustreError: 19948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 683.721562] LustreError: 19927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581006 with bad export cookie 9429784141738428291 [ 683.728987] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 683.736162] LustreError: 19927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 684.016956] Lustre: server umount lustre-MDT0001 complete [ 692.782287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.590893] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 705.752454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 710.478704] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 718.012382] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 724.198929] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 726.157555] Lustre: Failing over lustre-MDT0000 [ 726.172754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 726.179116] Lustre: Skipped 2 previous similar messages [ 726.438425] Lustre: server umount lustre-MDT0000 complete [ 745.648036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 745.840828] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 745.952174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 745.960633] Lustre: Skipped 5 previous similar messages [ 746.174307] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 750.179114] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 750.198525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 750.206357] Lustre: Skipped 1 previous similar message [ 750.227076] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 751.028044] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 761.435609] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 771.657096] Lustre: server umount lustre-MDT0000 complete [ 775.510807] LustreError: 24858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581098 with bad export cookie 9429784141738429229 [ 775.525515] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 775.531936] LustreError: 24858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 775.924055] Lustre: server umount lustre-MDT0001 complete [ 785.583084] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 00:05:06 (1788581106) [ 803.773761] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 814.652992] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 859.469471] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 872.935657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 873.348260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 873.386096] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 873.478610] Lustre: lustre-MDT0000: new disk, initializing [ 873.652176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 873.655129] Lustre: Skipped 3 previous similar messages [ 873.675793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 879.787555] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 893.897753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 894.007656] Lustre: 29675:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 894.288635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 899.623955] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 911.686467] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 917.848876] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 925.525031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 925.781677] Lustre: 31644:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 927.614941] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 927.673838] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 927.789794] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 927.818768] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 932.971724] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 946.051792] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 952.628049] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 978.915198] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 978.926569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 978.934948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 978.964071] Lustre: Skipped 3 previous similar messages [ 978.985596] Lustre: Skipped 5 previous similar messages [ 981.637531] Lustre: server umount lustre-OST0000 complete [ 986.595500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 986.603378] LustreError: Skipped 1 previous similar message [ 992.653676] Lustre: server umount lustre-MDT0000 complete [ 994.285068] LustreError: 29687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 994.311222] LustreError: 29687:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 996.850532] LustreError: 29666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581319 with bad export cookie 9429784141738429635 [ 996.872940] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 997.406656] Lustre: server umount lustre-MDT0001 complete [ 1010.506290] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 1013.483840] Key type lgssc unregistered [ 1013.804443] LNet: 33625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.824386] LNetError: 33625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.851508] LNet: Removed LNI 192.168.202.107@tcp [ 1014.822163] Key type .llcrypt unregistered [ 1014.826109] Key type ._llcrypt unregistered [ 1040.401758] Key type ._llcrypt registered [ 1040.407771] Key type .llcrypt registered [ 1040.525682] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 1041.493080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1041.517673] alg: No test for adler32 (adler32-zlib) [ 1042.688340] Lustre: Lustre: Build Version: 2.17.57_103_g3d8212a [ 1042.875713] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 1044.511172] Key type lgssc registered [ 1045.854327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1054.418159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1054.443701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1055.976873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1061.814549] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1065.445314] LustreError: 35035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1065.468137] LustreError: 35035:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1070.564807] LustreError: 35034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1072.542443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1073.058175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1079.042195] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1089.510294] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1096.200490] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1103.507757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1103.833983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1110.365331] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1117.179783] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1119.436075] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1125.383965] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1138.688874] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 00:10:59 (1788581459) [ 1140.763164] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1142.817514] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1150.269594] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 00:11:11 (1788581471) [ 1159.758731] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 00:11:20 (1788581480) [ 1160.789597] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1170.315197] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1170.324447] Lustre: Skipped 2 previous similar messages [ 1172.234949] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 00:11:32 (1788581492) [ 1176.323978] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1176.333422] Lustre: Skipped 3 previous similar messages [ 1196.532982] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1196.537724] Lustre: Skipped 6 previous similar messages [ 1199.830473] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 00:12:00 (1788581520) [ 1203.332564] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1207.387679] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1218.212501] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 00:12:18 (1788581538) [ 1220.492392] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1220.496635] Lustre: Skipped 3 previous similar messages [ 1235.414316] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 00:12:36 (1788581556) [ 1253.356448] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1253.358377] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1253.359438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1253.379509] Lustre: Skipped 1 previous similar message [ 1258.469709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1258.488570] Lustre: Skipped 2 previous similar messages [ 1259.480782] Lustre: server umount lustre-OST0000 complete [ 1263.585935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1263.589376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1263.601104] Lustre: Skipped 1 previous similar message [ 1263.609865] Lustre: Skipped 1 previous similar message [ 1268.710973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1269.837725] Lustre: server umount lustre-MDT0000 complete [ 1273.825124] LustreError: 35029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.846769] LustreError: 35029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1274.542413] LustreError: 35014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581597 with bad export cookie 3187858168693883797 [ 1274.552381] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1274.564464] LustreError: 35014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1275.020350] Lustre: server umount lustre-MDT0001 complete [ 1290.160893] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 1293.488776] Key type lgssc unregistered [ 1294.132199] LNet: 42893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1294.140188] LNetError: 42893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1294.199845] LNet: Removed LNI 192.168.202.107@tcp [ 1295.639243] Key type .llcrypt unregistered [ 1295.642471] Key type ._llcrypt unregistered [ 1318.362793] Key type ._llcrypt registered [ 1318.365206] Key type .llcrypt registered [ 1318.470620] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 1337.193679] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 1338.666067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1338.683641] alg: No test for adler32 (adler32-zlib) [ 1339.798787] Lustre: Lustre: Build Version: 2.17.57_103_g3d8212a [ 1340.106686] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 1341.815166] Key type lgssc registered [ 1342.860903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1383.808286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1383.898830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1385.166385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1385.198380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1385.297302] Lustre: lustre-MDT0000: new disk, initializing [ 1385.395687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1385.414957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1390.528393] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1402.035931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1402.153468] Lustre: 46051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1402.216507] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1402.220914] Lustre: Skipped 1 previous similar message [ 1402.341757] Lustre: lustre-MDT0001: new disk, initializing [ 1402.458947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1402.500169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1402.516486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1407.405215] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1418.520224] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1425.175592] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1433.195349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1433.655893] Lustre: lustre-OST0000: new disk, initializing [ 1433.665781] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1433.678128] Lustre: 48018:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1433.776593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1441.729645] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1441.857692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1441.874241] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1442.056444] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1453.103934] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1458.048371] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1458.263578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1464.092809] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1464.274425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1467.875914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1467.888764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1467.899553] Lustre: Skipped 1 previous similar message [ 1467.949065] Lustre: Skipped 1 previous similar message [ 1472.737247] Lustre: server umount lustre-OST0000 complete [ 1478.113908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1478.124451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1478.148015] Lustre: Skipped 1 previous similar message [ 1478.173133] Lustre: Skipped 1 previous similar message [ 1483.232482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1483.255470] Lustre: Skipped 1 previous similar message [ 1484.074825] Lustre: server umount lustre-MDT0000 complete [ 1488.881040] LustreError: 46041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581811 with bad export cookie 9511542422056000953 [ 1488.887900] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1488.901786] LustreError: 46041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1489.548888] Lustre: server umount lustre-MDT0001 complete [ 1503.981586] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 00:17:03 (1788581823) [ 1511.714409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1512.046613] LustreError: 50482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1512.058403] LustreError: 50482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1512.135235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1517.539579] LustreError: 50483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1518.734408] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1522.665441] LustreError: 50482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1527.777785] LustreError: 50483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1528.388511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1528.840073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1534.868996] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1544.452778] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1551.066129] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1559.500911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1559.994035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1567.525076] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1577.524472] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1587.044987] LustreError: 53011:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1588.156996] Lustre: Modifying parameter general.timeout=1 in log params [ 1590.089668] Lustre: Modifying parameter general.timeout=3 in log params [ 1590.095023] Lustre: Skipped 1 previous similar message [ 1592.648313] Lustre: Modifying parameter general.timeout=6 in log params [ 1592.653702] Lustre: Skipped 2 previous similar messages [ 1596.805828] Lustre: Modifying parameter general.timeout=10 in log params [ 1596.812220] Lustre: Skipped 3 previous similar messages [ 1604.961345] Lustre: Modifying parameter general.timeout=17 in log params [ 1604.966881] Lustre: Skipped 6 previous similar messages [ 1621.648991] Lustre: Modifying parameter general.timeout=30 in log params [ 1621.653614] Lustre: Skipped 12 previous similar messages [ 1653.677553] Lustre: Modifying parameter general.timeout=62 in log params [ 1653.681819] Lustre: Skipped 31 previous similar messages [ 1718.096096] Lustre: Modifying parameter general.timeout=124 in log params [ 1718.104748] Lustre: Skipped 61 previous similar messages [ 1735.075952] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 00:20:55 (1788582055) [ 1738.009453] Lustre: 59969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1739.751708] Lustre: 60078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1742.993890] Lustre: 60274:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1743.006847] Lustre: 60274:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1774.515738] Lustre: 61842:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1774.523762] Lustre: 61842:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1774.557611] LustreError: 61842:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1779.482143] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 00:21:39 (1788582099) [ 1795.044950] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1795.051242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1795.055444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1795.081967] Lustre: Skipped 1 previous similar message [ 1798.366832] Lustre: server umount lustre-MDT0000 complete [ 1800.162268] LustreError: 50478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1800.182985] LustreError: 50478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1802.003326] LustreError: 50464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582124 with bad export cookie 9511542422056001835 [ 1802.010430] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1802.014969] LustreError: 50464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1802.426543] Lustre: server umount lustre-MDT0001 complete [ 1817.103372] Lustre: server umount lustre-OST0000 complete [ 1844.930581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1845.229557] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1845.651832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1850.285377] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1858.353329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1858.490432] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1858.537553] Lustre: 64372:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1863.832945] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1872.190423] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1878.755054] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1885.787585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1885.963971] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1886.177658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1886.185379] Lustre: Skipped 1 previous similar message [ 1893.843084] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1901.552950] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1903.204200] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1903.217920] Lustre: Skipped 29 previous similar messages [ 1903.271697] Lustre: 66939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1912.291203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1912.294301] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1912.295829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1912.295835] Lustre: Skipped 2 previous similar messages [ 1912.307168] LustreError: Skipped 1 previous similar message [ 1912.353241] Lustre: Skipped 2 previous similar messages [ 1922.535479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1922.541367] Lustre: Skipped 4 previous similar messages [ 1924.063078] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1924.289733] Lustre: server umount lustre-OST0000 complete [ 1930.721857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1930.737417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1934.241716] Lustre: server umount lustre-MDT0000 complete [ 1937.856360] LustreError: 64366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582260 with bad export cookie 9511542422056005006 [ 1937.862904] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1937.868445] LustreError: 64366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1938.120526] Lustre: server umount lustre-MDT0001 complete [ 1947.728452] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 1950.193138] Key type lgssc unregistered [ 1950.458874] LNet: 68147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1950.466858] LNetError: 68147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1950.483982] LNet: Removed LNI 192.168.202.107@tcp [ 1951.303249] Key type .llcrypt unregistered [ 1951.305090] Key type ._llcrypt unregistered [ 1982.826441] Key type ._llcrypt registered [ 1982.830132] Key type .llcrypt registered [ 1982.959610] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 1984.109199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1984.377153] alg: No test for adler32 (adler32-zlib) [ 1985.525781] Lustre: Lustre: Build Version: 2.17.57_103_g3d8212a [ 1985.792383] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 1987.447307] Key type lgssc registered [ 1989.104438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1996.785826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1996.796429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1998.358059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.707400] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2008.546664] LustreError: 69561:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2008.565799] LustreError: 69561:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2013.342728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2013.774283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2017.846885] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2024.985418] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2030.424124] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2036.896238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2037.389314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2042.539550] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2049.181085] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2057.027584] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2069.473288] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2069.483408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2069.487492] Lustre: Skipped 1 previous similar message [ 2069.508187] Lustre: Skipped 2 previous similar messages [ 2072.252622] Lustre: server umount lustre-MDT0000 complete [ 2074.603334] LustreError: 70610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2074.626716] LustreError: 70610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2076.422821] LustreError: 69541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582399 with bad export cookie 2815842218665642762 [ 2076.435275] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2076.435711] LustreError: 69541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2076.803034] Lustre: server umount lustre-MDT0001 complete [ 2091.756850] Lustre: server umount lustre-OST0000 complete [ 2120.684169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2121.130086] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2121.205702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2121.546123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2126.921747] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2136.618525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.678452] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2136.706781] Lustre: 74432:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2136.772751] LustreError: 74442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2137.120075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2142.123704] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2150.722263] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2157.463838] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2164.793465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2165.053802] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2165.385389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2172.532039] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2179.944316] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2181.756966] Lustre: 77006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2183.846479] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2191.335258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2191.337232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2191.352111] Lustre: Skipped 1 previous similar message [ 2191.362635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2195.969225] Lustre: server umount lustre-OST0000 complete [ 2201.572984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2201.578459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2201.606756] Lustre: Skipped 1 previous similar message [ 2201.636927] Lustre: Skipped 2 previous similar messages [ 2206.690858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2206.708117] Lustre: Skipped 1 previous similar message [ 2211.809164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2211.828180] Lustre: Skipped 1 previous similar message [ 2214.367150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2214.567806] Lustre: server umount lustre-MDT0000 complete [ 2216.928182] LustreError: 74443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2216.942776] LustreError: 74443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2218.313430] LustreError: 74423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582541 with bad export cookie 2815842218665643609 [ 2218.319732] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2218.337898] LustreError: 74423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2218.689386] Lustre: server umount lustre-MDT0001 complete [ 2230.242273] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 2233.238415] Key type lgssc unregistered [ 2233.563743] LNet: 78214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2233.573470] LNetError: 78214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2233.591867] LNet: Removed LNI 192.168.202.107@tcp [ 2234.391261] Key type .llcrypt unregistered [ 2234.392501] Key type ._llcrypt unregistered [ 2261.729074] Key type ._llcrypt registered [ 2261.732188] Key type .llcrypt registered [ 2261.874221] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 2262.943744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2263.068398] alg: No test for adler32 (adler32-zlib) [ 2264.112338] Lustre: Lustre: Build Version: 2.17.57_103_g3d8212a [ 2264.378314] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 2266.081973] Key type lgssc registered [ 2267.347962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2275.957712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2275.989412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2277.445718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2281.314578] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2287.586186] LustreError: 79625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2287.601846] LustreError: 79625:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2289.735965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2290.594281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2295.374917] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2303.379909] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2309.508868] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2316.791860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2317.375957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2323.985869] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2331.233267] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2341.877851] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2342.910832] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2344.068851] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2346.148746] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2346.154411] Lustre: Skipped 1 previous similar message [ 2350.234259] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2350.241945] Lustre: Skipped 3 previous similar messages [ 2359.185850] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2359.194332] Lustre: Skipped 8 previous similar messages [ 2376.136370] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2376.140507] Lustre: Skipped 15 previous similar messages [ 2408.371644] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2408.376829] Lustre: Skipped 28 previous similar messages [ 2464.727189] Lustre: 87741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2468.516418] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 00:33:08 (1788582788) [ 2471.495502] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2473.971563] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 00:33:14 (1788582794) [ 2479.123826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.139431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2484.193880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2484.205121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2484.217668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2489.318202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2489.333330] Lustre: Skipped 2 previous similar messages [ 2493.407669] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2493.801193] Lustre: server umount lustre-OST0000 complete [ 2499.555041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2499.569958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2499.575747] Lustre: Skipped 1 previous similar message [ 2499.588158] Lustre: Skipped 1 previous similar message [ 2504.672729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2512.351186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2512.591291] Lustre: server umount lustre-MDT0000 complete [ 2514.913358] LustreError: 82130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2514.923314] LustreError: 82130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2516.689486] LustreError: 79604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582839 with bad export cookie 16459025265609818760 [ 2516.697341] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2516.712309] LustreError: 79604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2517.173321] Lustre: server umount lustre-MDT0001 complete [ 2533.746865] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 2536.676907] Key type lgssc unregistered [ 2536.988806] LNet: 88993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2536.994952] LNetError: 88993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2537.007369] LNet: Removed LNI 192.168.202.107@tcp [ 2537.956428] Key type .llcrypt unregistered [ 2537.959833] Key type ._llcrypt unregistered [ 2543.898251] Key type ._llcrypt registered [ 2543.901932] Key type .llcrypt registered [ 2544.060937] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2550.208844] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules [ 2551.451574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2551.478420] alg: No test for adler32 (adler32-zlib) [ 2552.698919] Lustre: Lustre: Build Version: 2.17.57_103_g3d8212a [ 2553.049333] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 2554.855848] Key type lgssc registered [ 2556.693338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2558.119347] LustreError: 90228:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2576.642255] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 2580.970812] LustreError: 90228:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2585.493188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2585.517278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2587.035104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2591.381575] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2596.325802] LustreError: 91264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.346316] LustreError: 91264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2600.043728] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 00:35:21 (1788582921) [ 2601.445379] LustreError: 91265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.312181] LustreError: 91296:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2604.321615] LustreError: 91296:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2604.326907] LustreError: 91296:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2604.549149] Lustre: server umount lustre-MDT0000 complete [ 2618.484345] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 2621.198332] Key type lgssc unregistered [ 2621.459984] LNet: 92787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2621.466900] LNetError: 92787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2621.492801] LNet: Removed LNI 192.168.202.107@tcp [ 2622.339972] Key type .llcrypt unregistered [ 2622.343972] Key type ._llcrypt unregistered [ 2643.470665] Key type ._llcrypt registered [ 2643.479576] Key type .llcrypt registered [ 2643.656450] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 2644.972041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2645.108924] alg: No test for adler32 (adler32-zlib) [ 2646.145188] Lustre: Lustre: Build Version: 2.17.57_103_g3d8212a [ 2646.394326] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 2648.039432] Key type lgssc registered [ 2649.432237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2657.033290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2657.050215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2658.633836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2663.573868] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2669.029378] LustreError: 94003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2669.058417] LustreError: 94003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2673.303318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2674.000272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2680.058166] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2690.734789] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2696.632409] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2704.535520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2705.584907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2711.956565] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2719.354851] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2778.884805] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 00:38:19 (1788583099) [ 2780.422712] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2782.231673] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 00:38:23 (1788583103) [ 2786.275773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2786.281437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2786.306812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2789.857151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2789.859300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2789.867376] Lustre: Skipped 1 previous similar message [ 2789.875642] Lustre: Skipped 2 previous similar messages [ 2792.730907] Lustre: server umount lustre-MDT0000 complete [ 2794.981854] LustreError: 94004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2796.444333] LustreError: 93983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583119 with bad export cookie 14924790860801252111 [ 2796.446725] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2796.458116] LustreError: 93983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2796.982289] Lustre: server umount lustre-MDT0001 complete [ 2811.878876] Lustre: server umount lustre-OST0000 complete [ 2825.656791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2826.228613] LustreError: 98320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2826.250566] LustreError: 98320:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2826.500511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2831.845101] LustreError: 98321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2832.932540] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2836.962990] LustreError: 98320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2843.132363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2843.932878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2849.213609] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2857.224530] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2863.469965] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2879.451735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2879.632082] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2879.640373] LustreError: 98308:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2879.657414] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2879.663251] LustreError: 100406:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2879.668515] LustreError: 100406:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2879.672162] LustreError: 100406:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2879.718798] Lustre: server umount lustre-OST0000 complete [ 2879.721631] LustreError: 100406:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2889.653838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2890.007074] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2890.016081] LustreError: 98308:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2890.044091] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2890.051230] LustreError: 100867:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2890.057176] LustreError: 100867:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2890.061349] LustreError: 100867:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2890.165232] Lustre: server umount lustre-OST0000 complete [ 2890.172293] LustreError: 100867:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2901.038365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2901.238945] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2901.242197] Lustre: Found index 0 for lustre-OST0000, updating log [ 2901.247877] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2901.353606] Lustre: lustre-OST0000: new disk, initializing [ 2901.362569] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2901.375887] Lustre: 101409:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2901.686077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2909.290247] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2913.718257] Lustre: Failing over lustre-OST0000 [ 2913.887603] Lustre: server umount lustre-OST0000 complete [ 2919.922551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2919.929926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2919.939941] Lustre: Skipped 1 previous similar message [ 2919.963374] Lustre: Skipped 1 previous similar message [ 2925.024991] LustreError: 98320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2925.046194] LustreError: 98320:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2926.408824] Lustre: server umount lustre-MDT0000 complete [ 2933.020664] LustreError: 98300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583255 with bad export cookie 14924790860801254855 [ 2933.023967] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2933.026944] LustreError: 98300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2933.389765] Lustre: server umount lustre-MDT0001 complete [ 2944.584740] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 00:41:04 (1788583264) [ 2964.279917] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 2976.903734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2977.254576] LustreError: 103756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2977.275822] LustreError: 103756:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2977.382622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2981.810889] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2990.468101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2996.694838] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2999.808945] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3007.438806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3015.896187] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3025.061954] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3025.155300] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3025.242138] Lustre: lustre-OST0001: new disk, initializing [ 3025.246675] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3025.250648] Lustre: 106076:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3025.511191] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3025.515700] Lustre: Skipped 2 previous similar messages [ 3032.262618] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3034.261516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3034.275563] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3034.383108] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3041.089859] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3048.006294] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3055.080495] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3055.086628] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3055.094433] Lustre: Skipped 2 previous similar messages [ 3055.103705] Lustre: Skipped 2 previous similar messages [ 3060.206527] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3060.217271] Lustre: Skipped 3 previous similar messages [ 3067.359497] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3067.575208] Lustre: server umount lustre-MDT0001 complete [ 3070.438155] LustreError: 104488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3070.458816] LustreError: 104488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3079.771221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3079.942457] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3079.961836] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3080.024826] Lustre: 103743:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3081.762190] LustreError: 107617:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3081.779069] LustreError: 107617:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3081.789227] LustreError: 103747:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3081.919736] LustreError: 107623:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.107@tcp: cfg command failed: rc = -17 [ 3081.930528] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3081.930528] [ 3086.327593] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3087.830233] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3099.736403] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 00:43:39 (1788583419) [ 3121.943990] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 3141.614866] Lustre: 109694:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3167.933816] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3172.309049] Lustre: 110829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3263.457663] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3263.460265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3263.463310] Lustre: Skipped 5 previous similar messages [ 3263.505566] Lustre: Skipped 1 previous similar message [ 3264.258986] Lustre: server umount lustre-MDT0000 complete [ 3268.592332] LustreError: 104498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3268.628737] LustreError: 104498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3269.133163] LustreError: 106075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583592 with bad export cookie 14924790860801255842 [ 3269.143519] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3269.150196] LustreError: 106075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3269.606591] Lustre: server umount lustre-MDT0001 complete [ 3285.079048] Lustre: server umount lustre-OST0000 complete [ 3289.055371] Lustre: 93314:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788583596/real 1788583596] req@ffff9b583840f480 x1875465175680896/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788583612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3289.086717] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3289.103000] Lustre: Skipped 2 previous similar messages [ 3289.484619] Lustre: server umount lustre-OST0001 complete [ 3296.513562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3308.657605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3318.454167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3330.512862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3353.550978] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 3364.346493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3364.389688] Lustre: lustre-MDT0000: reset Object Index mappings [ 3365.114443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3365.119085] Lustre: Skipped 1 previous similar message [ 3370.322700] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3380.632464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3380.651943] Lustre: lustre-MDT0001: reset Object Index mappings [ 3386.826444] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3390.503775] Lustre: 115934:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3398.586867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3406.202646] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3406.765226] LustreError: 116288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3406.781404] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3406.786707] LustreError: 116288:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3409.862781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3417.561649] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3423.754577] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3423.757519] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3425.833629] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3435.439881] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3439.774843] Lustre: 117811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3497.308829] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 00:50:17 (1788583817) [ 3524.579550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3524.592285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3524.614586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3524.620922] Lustre: Skipped 3 previous similar messages [ 3527.681635] Lustre: server umount lustre-MDT0000 complete [ 3531.501642] LustreError: 115936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583854 with bad export cookie 14924790860801302742 [ 3531.514161] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3531.527180] LustreError: 115936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3538.344359] Lustre: server umount lustre-MDT0001 complete [ 3549.642920] Lustre: server umount lustre-OST0000 complete [ 3560.377527] Lustre: server umount lustre-OST0001 complete [ 3568.729760] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 3582.262959] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 3619.096517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3619.398567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3619.436932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3619.506572] Lustre: lustre-MDT0000: new disk, initializing [ 3619.593611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3619.598920] Lustre: Skipped 3 previous similar messages [ 3619.613658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3623.839840] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3630.372608] Lustre: server umount lustre-MDT0000 complete [ 3640.588434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3640.867139] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3640.878475] Lustre: 122099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3640.887602] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3645.994900] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3651.073570] Lustre: server umount lustre-MDT0000 complete [ 3661.319767] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 00:53:01 (1788583981) [ 3663.097859] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3665.262449] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 00:53:05 (1788583985) [ 3684.729749] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 3695.378091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3700.196244] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3708.935684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3709.056662] Lustre: 124149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3709.116549] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3709.121242] Lustre: Skipped 1 previous similar message [ 3709.203208] Lustre: lustre-MDT0001: new disk, initializing [ 3709.317524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3709.326875] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3714.317922] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3719.197170] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3725.734937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3726.004239] Lustre: lustre-OST0000: new disk, initializing [ 3726.010903] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3726.018233] Lustre: 125730:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3727.630139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3727.646917] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3727.767583] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3732.605525] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3743.194878] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3743.365150] Lustre: lustre-OST0001: new disk, initializing [ 3743.368437] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3743.373821] Lustre: 126601:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3745.189181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3745.195707] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3745.249650] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3750.347827] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3759.470509] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3764.003413] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4166.375676] Lustre: 133022:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4166.389771] Lustre: 133022:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4166.401834] Lustre: 133022:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4166.418632] Lustre: 133022:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4166.430895] Lustre: 133022:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4166.443375] Lustre: 133022:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4463.434790] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:06:23 (1788584783) [ 4482.534525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4482.539395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4482.562159] Lustre: Skipped 5 previous similar messages [ 4482.564469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4482.591907] Lustre: Skipped 9 previous similar messages [ 4484.766625] Lustre: server umount lustre-MDT0000 complete [ 4487.661728] LustreError: 127413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4487.686210] LustreError: 127413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 4489.214661] LustreError: 124889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584812 with bad export cookie 14924790860801372056 [ 4489.220693] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4489.237122] LustreError: 124889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4492.772536] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4492.785246] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4492.786449] Lustre: Skipped 4 previous similar messages [ 4492.826516] Lustre: Skipped 1 previous similar message [ 4495.910771] Lustre: server umount lustre-MDT0001 complete [ 4515.295305] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4515.659455] Lustre: server umount lustre-OST0000 complete [ 4535.263132] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4535.571622] Lustre: server umount lustre-OST0001 complete [ 4543.283304] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 4555.886544] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 4598.095528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4598.363286] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4598.392722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4598.517561] Lustre: lustre-MDT0000: new disk, initializing [ 4598.674768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4598.688640] Lustre: Skipped 5 previous similar messages [ 4598.714665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4603.128157] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 4613.932217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4614.100304] Lustre: 136086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4614.161981] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4614.165182] Lustre: Skipped 1 previous similar message [ 4614.252261] Lustre: lustre-MDT0001: new disk, initializing [ 4614.379397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4614.395698] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4619.726653] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 4630.421386] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4637.673996] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4647.089590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4647.522352] Lustre: lustre-OST0000: new disk, initializing [ 4647.524799] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4647.527987] Lustre: 138053:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4647.668970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4647.677499] Lustre: Skipped 1 previous similar message [ 4649.384093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4649.405142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4649.639564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4655.896734] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 4666.791507] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4679.119896] Lustre: lustre-MDD0000: changelog on [ 4682.368938] Lustre: lustre-MDD0001: changelog on [ 6631.901498] Lustre: lustre-MDD0001: changelog off [ 6633.847286] Lustre: 136092:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6634.691408] Lustre: lustre-MDD0000: changelog off [ 6641.122353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6641.125846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6641.129296] LustreError: Skipped 1 previous similar message [ 6641.130283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6641.148141] Lustre: Skipped 1 previous similar message [ 6641.162774] Lustre: Skipped 3 previous similar messages [ 6644.195651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6644.202073] Lustre: Skipped 1 previous similar message [ 6646.765897] Lustre: server umount lustre-OST0000 complete [ 6652.384948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6652.396202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6652.429183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6654.439798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6657.248577] Lustre: server umount lustre-MDT0000 complete [ 6659.553507] LustreError: 136096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6659.567374] LustreError: 136096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6660.882321] LustreError: 136077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586983 with bad export cookie 14924790860802192792 [ 6660.883875] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6660.897885] LustreError: 136077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6661.140746] Lustre: server umount lustre-MDT0001 complete [ 6673.229899] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 6675.541695] Key type lgssc unregistered [ 6675.812182] LNet: 147187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6675.822610] LNetError: 147187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6675.840715] LNet: Removed LNI 192.168.202.107@tcp [ 6676.680184] Key type .llcrypt unregistered [ 6676.682074] Key type ._llcrypt unregistered [ 6696.226870] Key type ._llcrypt registered [ 6696.228638] Key type .llcrypt registered [ 6696.373988] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 6714.024930] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 6714.916515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6715.045080] alg: No test for adler32 (adler32-zlib) [ 6716.120785] Lustre: Lustre: Build Version: 2.17.57_103_g3d8212a [ 6716.449470] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 6718.143167] Key type lgssc registered [ 6719.438722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6756.683585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6756.708766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.974122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6758.006838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6758.108430] Lustre: lustre-MDT0000: new disk, initializing [ 6758.240460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6758.264184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6762.739302] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6773.880894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6774.019288] Lustre: 150383:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6774.077870] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6774.082838] Lustre: Skipped 1 previous similar message [ 6774.154837] Lustre: lustre-MDT0001: new disk, initializing [ 6774.256490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6774.299871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6774.307293] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6778.703959] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6788.971763] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6795.355574] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6802.769960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6802.941428] Lustre: lustre-OST0000: new disk, initializing [ 6802.947804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6802.953134] Lustre: 152351:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6803.045795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6809.501551] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6810.759318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6810.769771] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6810.873178] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6819.623607] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6825.189120] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_module obdecho/obdecho [ 6827.270454] LustreError: 153353:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6827.280920] LustreError: 153353:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6829.140605] LustreError: 153450:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6841.828322] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6841.833666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6841.844252] Lustre: Skipped 2 previous similar messages [ 6841.859067] Lustre: Skipped 2 previous similar messages [ 6846.950879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6846.979167] Lustre: Skipped 2 previous similar messages [ 6847.184500] Lustre: server umount lustre-MDT0000 complete [ 6851.636674] LustreError: 151189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587174 with bad export cookie 2142759446811447941 [ 6851.652110] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6851.665143] LustreError: 151189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6852.044638] Lustre: server umount lustre-MDT0001 complete [ 6867.198838] Lustre: server umount lustre-OST0000 complete [ 6891.024737] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 6903.562411] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 6944.460598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6944.670922] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6944.693733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6944.758428] Lustre: lustre-MDT0000: new disk, initializing [ 6944.823819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6944.837373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6949.313933] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6958.886420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6958.980531] Lustre: 157168:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6959.033217] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6959.040844] Lustre: Skipped 1 previous similar message [ 6959.116867] Lustre: lustre-MDT0001: new disk, initializing [ 6959.211056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6959.255252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6959.276500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6962.956233] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6972.894541] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6979.380491] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6988.437627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6988.737905] Lustre: lustre-OST0000: new disk, initializing [ 6988.741851] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6988.749927] Lustre: 159139:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6988.843222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6994.938796] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6998.087724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6998.095624] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6998.216254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7004.075414] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7008.588108] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7008.814419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7012.937192] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7013.140736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7018.980095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7018.984683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7018.987249] LustreError: Skipped 1 previous similar message [ 7019.001826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7019.004583] Lustre: Skipped 1 previous similar message [ 7019.018916] Lustre: Skipped 1 previous similar message [ 7024.104080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7024.108038] Lustre: Skipped 1 previous similar message [ 7029.215195] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7029.220234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7029.235670] Lustre: Skipped 1 previous similar message [ 7029.390810] Lustre: server umount lustre-OST0000 complete [ 7034.343943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7034.352354] Lustre: Skipped 1 previous similar message [ 7039.284804] Lustre: server umount lustre-MDT0000 complete [ 7043.104418] LustreError: 157161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587366 with bad export cookie 2142759446811448900 [ 7043.111338] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7043.118154] LustreError: 157161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7043.491212] Lustre: server umount lustre-MDT0001 complete [ 7047.068270] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 01:49:27 (1788587367) [ 7048.798872] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7050.852807] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 01:49:31 (1788587371) [ 7057.692176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7058.037401] LustreError: 161551:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7058.046943] LustreError: 161551:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7058.121891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7062.641179] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7063.527160] LustreError: 161552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7068.657156] LustreError: 161551:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7071.506411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7076.697276] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7084.484355] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7091.051243] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7098.049658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7098.321081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7098.323836] Lustre: Skipped 1 previous similar message [ 7106.696909] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7115.415308] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7126.549667] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.7@tcp (stopping) [ 7126.551772] Lustre: Skipped 2 previous similar messages [ 7127.011591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7127.019657] Lustre: Skipped 2 previous similar messages [ 7127.665966] Lustre: server umount lustre-MDT0000 complete [ 7131.660569] LustreError: 161547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7132.195505] LustreError: 161531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587455 with bad export cookie 2142759446811449782 [ 7132.203804] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7136.785330] LustreError: 161547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7136.808207] LustreError: 161547:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7137.250920] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7138.667670] Lustre: server umount lustre-MDT0001 complete [ 7145.801403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7159.557660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7159.807277] LustreError: 165392:0:(llog.c:1655:llog_backup()) MGC192.168.202.107@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7159.813345] Lustre: 165392:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.107@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7167.559062] LustreError: 165397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7167.576457] LustreError: 165397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7183.877139] LustreError: 165398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7183.903197] LustreError: 165398:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 7217.695748] LustreError: 165416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7217.725838] LustreError: 165416:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 23 previous similar messages [ 7220.704829] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7220.716404] Lustre: 165392:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.107@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7261.699723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7265.981064] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7266.799277] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7276.078969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7277.219994] LustreError: 166136:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 7277.223240] LustreError: 166136:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@0000000046531b28 x1875469443132288/t0(0) o250->MGC192.168.202.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788587600 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7277.237566] LustreError: 166136:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 7278.066217] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1dbc9c5d324f32ec [ 7278.085711] LustreError: 166145:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.202.107@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000004db2e752) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7278.107785] Lustre: MGC192.168.202.107@tcp: Connection restored to 0@lo (at 0@lo) [ 7283.083640] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7283.704266] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7291.645205] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7293.675797] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7299.962585] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7302.118506] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7324.644445] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7324.662284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7324.662785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7324.666506] Lustre: Skipped 2 previous similar messages [ 7324.696281] Lustre: Skipped 7 previous similar messages [ 7325.440696] Lustre: server umount lustre-MDT0000 complete [ 7329.304175] LustreError: 161531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587652 with bad export cookie 2142759446811456236 [ 7329.318719] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7329.320514] LustreError: 161531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7329.816973] Lustre: server umount lustre-MDT0001 complete [ 7344.437548] Lustre: server umount lustre-OST0000 complete [ 7369.277741] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 7383.435806] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 7423.630471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7423.894121] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7423.948316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7424.182836] Lustre: lustre-MDT0000: new disk, initializing [ 7424.321534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7424.324472] Lustre: Skipped 1 previous similar message [ 7424.342477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7429.051762] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7440.649939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7440.749114] Lustre: 170498:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7440.998884] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7445.362961] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7455.865517] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7462.322980] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7469.637184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7469.875952] Lustre: lustre-OST0000: new disk, initializing [ 7469.880177] Lustre: Skipped 1 previous similar message [ 7469.887780] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7469.893280] Lustre: Skipped 2 previous similar messages [ 7469.899309] Lustre: 172467:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7475.787317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7475.798268] Lustre: Skipped 1 previous similar message [ 7475.815324] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7475.927706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7476.297573] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7486.331841] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7491.672956] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7491.915674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7496.883640] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7497.058536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7501.796336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7501.804445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7501.805594] LustreError: Skipped 1 previous similar message [ 7501.808423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7501.818863] Lustre: Skipped 1 previous similar message [ 7501.838418] Lustre: Skipped 1 previous similar message [ 7505.332021] Lustre: server umount lustre-OST0000 complete [ 7515.652204] Lustre: server umount lustre-MDT0000 complete [ 7517.157911] LustreError: 170509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7517.171404] LustreError: 170509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 46 previous similar messages [ 7519.290654] LustreError: 170491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587842 with bad export cookie 2142759446811473078 [ 7519.308098] LustreError: 170491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7519.311418] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7519.654806] Lustre: server umount lustre-MDT0001 complete [ 7528.816596] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 01:57:29 (1788587849) [ 7535.215348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7540.415601] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7548.724857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7553.266356] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7560.920832] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7567.056604] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7574.226591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7581.839129] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7592.217405] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7608.782400] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 01:58:49 (1788587929) [ 7615.458844] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7615.470183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7615.481866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7615.498190] Lustre: Skipped 3 previous similar messages [ 7615.518419] Lustre: Skipped 3 previous similar messages [ 7619.237204] Lustre: server umount lustre-OST0000 complete [ 7625.697167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7630.990074] Lustre: server umount lustre-MDT0000 complete [ 7634.976889] LustreError: 174859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587957 with bad export cookie 2142759446811473953 [ 7634.994261] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7634.995086] LustreError: 174859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7635.363266] Lustre: server umount lustre-MDT0001 complete [ 7648.807377] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 7651.385789] Key type lgssc unregistered [ 7651.728032] LNet: 178610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7651.731915] LNetError: 178610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7651.747620] LNet: Removed LNI 192.168.202.107@tcp [ 7652.361678] Key type .llcrypt unregistered [ 7652.364347] Key type ._llcrypt unregistered [ 7674.140662] Key type ._llcrypt registered [ 7674.142765] Key type .llcrypt registered [ 7674.237903] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 7675.364587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7675.584685] alg: No test for adler32 (adler32-zlib) [ 7676.584638] Lustre: Lustre: Build Version: 2.17.57_103_g3d8212a [ 7676.800972] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 7678.447750] Key type lgssc registered [ 7679.665233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7687.372720] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7687.390385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7702.943482] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7708.063463] LustreError: 179860:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.107@tcp: failed processing log, type 4: rc = -110 [ 7713.183976] LustreError: 179860:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7713.197222] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7713.214130] LustreError: MGC192.168.202.107@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7713.220407] LustreError: 179860:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7713.225744] LustreError: 179860:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7713.230161] LustreError: 179860:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7713.281439] Lustre: server umount lustre-OST0000 complete [ 7713.285396] LustreError: 179860:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7722.957460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7724.283624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7730.640980] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7734.770845] LustreError: 180359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7734.794107] LustreError: 180359:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7739.879119] LustreError: 180360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7741.206469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7741.638153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7747.132525] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7756.140263] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7762.828715] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7770.317616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7770.478779] LustreError: 182188:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7770.486306] Lustre: 182188:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.107@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7770.633281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7778.290497] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7785.974313] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7792.106700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7792.120560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7792.124548] Lustre: Skipped 2 previous similar messages [ 7792.145286] Lustre: Skipped 2 previous similar messages [ 7794.693777] Lustre: server umount lustre-MDT0000 complete [ 7797.237985] LustreError: 181099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7797.261248] LustreError: 181099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7798.683682] LustreError: 180340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588121 with bad export cookie 408603923225481053 [ 7798.700356] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7799.056256] Lustre: server umount lustre-MDT0001 complete [ 7813.647427] Lustre: server umount lustre-OST0000 complete [ 7824.896323] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 7841.158738] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 7884.232738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7884.446785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7884.465557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7884.522686] Lustre: lustre-MDT0000: new disk, initializing [ 7884.575877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7884.590370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7888.859976] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7900.258030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7900.447626] Lustre: 185738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7900.567981] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7900.575541] Lustre: Skipped 1 previous similar message [ 7900.690255] Lustre: lustre-MDT0001: new disk, initializing [ 7900.790706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7900.828727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7900.849201] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7905.421200] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7915.198445] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7921.542744] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7928.627078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7928.846785] Lustre: lustre-OST0000: new disk, initializing [ 7928.850061] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7928.856814] Lustre: 187705:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7928.931549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7935.309440] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7935.594467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7935.600787] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7935.751334] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7944.731301] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7949.672833] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7949.937881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7954.223590] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7954.432979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7956.448188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7956.453478] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7956.465348] LustreError: Skipped 1 previous similar message [ 7956.474439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7956.498132] Lustre: Skipped 1 previous similar message [ 7956.531645] Lustre: Skipped 1 previous similar message [ 7961.571210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7961.587525] Lustre: Skipped 1 previous similar message [ 7962.620578] Lustre: server umount lustre-OST0000 complete [ 7966.689756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7966.730172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7966.735534] Lustre: Skipped 1 previous similar message [ 7971.811142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7971.818288] Lustre: Skipped 1 previous similar message [ 7973.048415] Lustre: server umount lustre-MDT0000 complete [ 7976.931515] LustreError: 185749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7976.970597] LustreError: 185749:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7977.457309] LustreError: 185730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588300 with bad export cookie 408603923225481767 [ 7977.463514] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7977.469129] LustreError: 185730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7977.850692] Lustre: server umount lustre-MDT0001 complete [ 7988.588242] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 02:05:09 (1788588309) [ 8007.588352] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 8010.491563] Key type lgssc unregistered [ 8010.720926] LNet: 190648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8010.739121] LNetError: 190648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8010.759672] LNet: Removed LNI 192.168.202.107@tcp [ 8011.959525] Key type .llcrypt unregistered [ 8011.962605] Key type ._llcrypt unregistered [ 8032.911692] Key type ._llcrypt registered [ 8032.918756] Key type .llcrypt registered [ 8033.099900] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 8034.132588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8034.254317] alg: No test for adler32 (adler32-zlib) [ 8035.752439] Lustre: Lustre: Build Version: 2.17.57_103_g3d8212a [ 8035.963405] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 8037.623220] Key type lgssc registered [ 8039.244423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8047.686299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8047.706110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8049.348555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8053.944962] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8059.881039] LustreError: 191864:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8059.900075] LustreError: 191864:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8062.653398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8063.041292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8067.870847] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8075.189925] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8080.769506] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8089.511741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8089.681143] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8089.683880] LustreError: 193727:0:(llog.c:1696:llog_backup()) MGC192.168.202.107@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8089.692396] Lustre: 193727:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.107@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8089.815922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8096.581473] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8105.763921] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8109.024399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8109.034389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8109.059802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8112.609449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8112.617916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8115.002750] Lustre: server umount lustre-MDT0000 complete [ 8117.745432] LustreError: 193486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8117.772312] LustreError: 193486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8119.541939] LustreError: 191845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588442 with bad export cookie 15256230432337708109 [ 8119.559504] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8119.998032] Lustre: server umount lustre-MDT0001 complete [ 8133.424848] Lustre: server umount lustre-OST0000 complete [ 8143.959880] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 8160.355494] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 8202.122316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8202.367521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8202.439225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8202.564380] Lustre: lustre-MDT0000: new disk, initializing [ 8202.637828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8202.687040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8208.080360] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8221.240624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8221.404197] Lustre: 197280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8221.457149] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8221.460062] Lustre: Skipped 1 previous similar message [ 8221.463996] LustreError: 198063:0:(llog.c:1696:llog_backup()) MGC192.168.202.107@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8221.474073] LustreError: 198063:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8221.482543] Lustre: 198063:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.107@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8221.512770] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8221.515422] Lustre: Skipped 1 previous similar message [ 8221.613060] Lustre: lustre-MDT0001: new disk, initializing [ 8221.724743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8221.772088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8221.788324] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8227.649362] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8238.365848] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8244.421562] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8252.358076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8252.623684] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8252.626066] Lustre: Skipped 1 previous similar message [ 8252.627746] LustreError: 199224:0:(llog.c:1696:llog_backup()) MGC192.168.202.107@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8252.632723] LustreError: 199224:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8252.645393] Lustre: 199224:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.107@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8252.661351] Lustre: 199224:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8252.678298] Lustre: lustre-OST0000: new disk, initializing [ 8252.685570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8252.697106] Lustre: 199247:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8252.782492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8259.741095] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8260.743355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8260.767759] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8260.939606] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8269.760788] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8275.126487] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8275.390800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8280.080264] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8280.419347] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8286.693037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8286.708778] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8286.710749] Lustre: Skipped 1 previous similar message [ 8286.712249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8286.712254] Lustre: Skipped 2 previous similar messages [ 8286.738644] LustreError: Skipped 1 previous similar message [ 8288.373077] Lustre: server umount lustre-OST0000 complete [ 8293.345833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8293.352812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8293.363603] Lustre: Skipped 1 previous similar message [ 8293.373519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8293.388749] Lustre: Skipped 1 previous similar message [ 8298.327688] Lustre: server umount lustre-MDT0000 complete [ 8301.899718] LustreError: 198075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588624 with bad export cookie 15256230432337708830 [ 8301.903602] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8301.908982] LustreError: 198075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8302.272857] Lustre: server umount lustre-MDT0001 complete [ 8311.801979] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 02:10:32 (1788588632) [ 8333.232330] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 8344.025041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8344.346184] LustreError: 202375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8344.453495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8349.331758] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8349.676651] LustreError: 202376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8354.791458] LustreError: 202375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8359.413044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8365.126931] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8369.066553] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8379.392087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8379.879582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8379.884957] Lustre: Skipped 1 previous similar message [ 8386.864424] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8395.882287] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8396.103485] Lustre: lustre-OST0001: new disk, initializing [ 8396.107534] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8396.113577] Lustre: 204684:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8398.878831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8398.900679] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8399.152410] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8403.363770] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8414.848230] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8425.715318] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8428.273666] Lustre: DEBUG MARKER: ADD OST3 [ 8433.901659] loop: module loaded [ 8433.984352] loop0: detected capacity change from 0 to 204800000 [ 8434.377187] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8434.413746] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8434.428390] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8441.124212] loop0: detected capacity change from 0 to 204800000 [ 8441.477176] Lustre: DEBUG MARKER: STOP OST3 [ 8446.893976] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8447.159920] Lustre: lustre-OST0003: new disk, initializing [ 8447.168779] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8447.175382] Lustre: 206680:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8447.302031] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8447.311933] Lustre: Skipped 1 previous similar message [ 8449.055696] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8449.221377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8449.238285] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8449.285460] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8451.103299] LustreError: 206858:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8451.407612] Lustre: Failing over lustre-OST0003 [ 8451.579635] Lustre: server umount lustre-OST0003 complete [ 8454.662376] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8454.667150] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8454.678898] LustreError: 203860:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8454.683870] Lustre: Skipped 2 previous similar messages [ 8454.701015] LustreError: 203860:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8455.521201] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8463.873359] Lustre: DEBUG MARKER: START OST3 again [ 8471.426656] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8471.954576] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8473.446830] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8473.650258] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8473.650572] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8478.841209] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8500.526164] Lustre: Failing over lustre-OST0003 [ 8500.626455] Lustre: server umount lustre-OST0003 complete [ 8502.759724] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8502.776100] LustreError: 205827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8502.790410] LustreError: 205827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 8518.122691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8518.130803] Lustre: Skipped 3 previous similar messages [ 8524.232205] Lustre: server umount lustre-MDT0000 complete [ 8527.846644] LustreError: 202355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588850 with bad export cookie 15256230432337709712 [ 8527.866208] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8527.882989] LustreError: 202355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8528.354134] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8528.376898] Lustre: Skipped 5 previous similar messages [ 8528.385569] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8528.397438] Lustre: Skipped 5 previous similar messages [ 8537.516518] LustreError: 207958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8537.532709] LustreError: 207958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 8542.175476] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8542.397194] Lustre: server umount lustre-MDT0001 complete [ 8546.295202] Lustre: server umount lustre-OST0000 complete [ 8550.505746] Lustre: server umount lustre-OST0001 complete [ 8557.220581] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 8567.879793] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 8613.952238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8614.283684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8614.314771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8614.406349] Lustre: lustre-MDT0000: new disk, initializing [ 8614.545255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8614.552338] Lustre: Skipped 1 previous similar message [ 8614.566911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8620.150290] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8631.153379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8631.261512] Lustre: 212146:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8631.307732] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8631.312152] Lustre: Skipped 1 previous similar message [ 8631.383935] Lustre: lustre-MDT0001: new disk, initializing [ 8631.506562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8631.515357] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8635.868530] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8647.686141] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8653.497917] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8660.495946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8660.837891] Lustre: 214117:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8662.393386] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8662.520754] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8666.528082] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8675.666615] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8679.722516] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8679.944132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8683.903619] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8684.086746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8688.097560] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8688.114274] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8688.126885] Lustre: Skipped 1 previous similar message [ 8688.132183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8688.141083] Lustre: Skipped 5 previous similar messages [ 8700.383397] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8700.598083] Lustre: server umount lustre-OST0000 complete [ 8708.582945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8708.589374] LustreError: Skipped 1 previous similar message [ 8711.206552] Lustre: server umount lustre-MDT0000 complete [ 8713.700412] LustreError: 212157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8713.718730] LustreError: 212157:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8714.760213] LustreError: 212137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589037 with bad export cookie 15256230432337712008 [ 8714.767339] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8714.773196] LustreError: 212137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8715.123561] Lustre: server umount lustre-MDT0001 complete [ 8719.241831] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 02:17:19 (1788589039) [ 8736.716206] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 8747.772713] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 8790.124996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8790.434479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8790.462870] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8790.467765] Lustre: Skipped 1 previous similar message [ 8790.540560] Lustre: lustre-MDT0000: new disk, initializing [ 8790.544914] Lustre: Skipped 1 previous similar message [ 8790.682260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8790.690679] Lustre: Skipped 1 previous similar message [ 8795.686055] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8806.136505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8806.319411] Lustre: 218731:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8806.550604] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8811.611651] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8822.582259] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8829.587943] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8837.470388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8837.865868] Lustre: 220702:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8839.788616] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8839.897796] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8844.344442] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8853.791841] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8858.667632] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8858.907708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8864.421440] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8864.714809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8870.883556] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8870.887810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8870.891980] LustreError: Skipped 1 previous similar message [ 8870.895292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8870.915047] Lustre: Skipped 4 previous similar messages [ 8870.934422] Lustre: Skipped 7 previous similar messages [ 8873.630346] Lustre: server umount lustre-OST0000 complete [ 8883.682607] Lustre: server umount lustre-MDT0000 complete [ 8886.241118] LustreError: 219609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8886.268250] LustreError: 219609:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8887.926138] LustreError: 218725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589210 with bad export cookie 15256230432337712890 [ 8887.933805] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8887.944389] LustreError: 218725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8888.422779] Lustre: server umount lustre-MDT0001 complete [ 8897.987431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8898.489357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8898.496755] Lustre: Skipped 5 previous similar messages [ 8902.719883] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8912.254690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8917.399589] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8925.849426] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8931.837282] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8938.856035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8947.668597] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8955.773541] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8968.162031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8968.174722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8968.179577] LustreError: Skipped 1 previous similar message [ 8968.186320] Lustre: Skipped 4 previous similar messages [ 8972.630891] Lustre: server umount lustre-OST0000 complete [ 8983.219596] Lustre: server umount lustre-MDT0000 complete [ 8986.935639] LustreError: 222899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589309 with bad export cookie 15256230432337713772 [ 8986.950537] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8987.199120] Lustre: server umount lustre-MDT0001 complete [ 9095.704927] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 9098.585882] Key type lgssc unregistered [ 9098.968703] LNet: 226583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9098.994528] LNetError: 226583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9099.021646] LNet: Removed LNI 192.168.202.107@tcp [ 9100.105439] Key type .llcrypt unregistered [ 9100.109914] Key type ._llcrypt unregistered [ 9133.929850] Key type ._llcrypt registered [ 9133.932515] Key type .llcrypt registered [ 9134.056829] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 9154.223264] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 9155.579724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9155.720898] alg: No test for adler32 (adler32-zlib) [ 9156.996679] Lustre: Lustre: Build Version: 2.17.57_103_g3d8212a [ 9157.191485] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 9158.840987] Key type lgssc registered [ 9160.303159] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9207.038519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9207.082276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9208.351589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9208.402412] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9208.519901] Lustre: lustre-MDT0000: new disk, initializing [ 9208.595560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9208.612602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9213.675776] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9225.519410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9225.653816] Lustre: 229934:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9225.712476] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9225.716347] Lustre: Skipped 1 previous similar message [ 9225.812679] Lustre: lustre-MDT0001: new disk, initializing [ 9225.938601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9225.975611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9225.991680] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9230.610174] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9241.878284] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9249.478477] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9258.008627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9258.339055] Lustre: lustre-OST0000: new disk, initializing [ 9258.342838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9258.355039] Lustre: 231903:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9258.474719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9265.689411] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9266.801181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9266.818522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9266.969529] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9276.450208] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9283.045348] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9283.405845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9289.689984] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9289.994385] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9292.781719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9292.784968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9292.788604] LustreError: Skipped 1 previous similar message [ 9292.790971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9292.796955] Lustre: Skipped 1 previous similar message [ 9292.810721] Lustre: Skipped 1 previous similar message [ 9297.893606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9297.902091] Lustre: Skipped 1 previous similar message [ 9298.643616] Lustre: server umount lustre-OST0000 complete [ 9303.009136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9303.032428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9303.038948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9303.063547] Lustre: Skipped 1 previous similar message [ 9303.090757] Lustre: Skipped 1 previous similar message [ 9308.131895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9308.141683] Lustre: Skipped 1 previous similar message [ 9313.251545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9313.277364] Lustre: Skipped 1 previous similar message [ 9317.343963] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9317.571884] Lustre: server umount lustre-MDT0000 complete [ 9321.831263] LustreError: 230729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589644 with bad export cookie 13874386052522608428 [ 9321.837038] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9321.840132] LustreError: 230729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9322.151709] Lustre: server umount lustre-MDT0001 complete [ 9347.555289] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 9360.125173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9360.568647] LustreError: 234861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9360.584847] LustreError: 234861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9360.637723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9365.559636] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9365.986523] LustreError: 234862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9370.098024] LustreError: 234861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9374.822297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9375.204181] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9375.345683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9380.737343] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9384.352646] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9391.586864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9392.122542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9399.770685] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9409.498722] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9409.631289] Lustre: lustre-OST0001: new disk, initializing [ 9409.633540] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9409.639112] Lustre: 237167:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9409.709445] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9416.650626] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9419.971743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9419.989381] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9420.177118] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9429.811194] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9438.819972] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9454.330713] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 02:29:34 (1788589774) [ 9461.223577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9461.239989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9461.255488] Lustre: Skipped 2 previous similar messages [ 9461.267777] Lustre: Skipped 3 previous similar messages [ 9463.962660] Lustre: server umount lustre-MDT0000 complete [ 9466.343700] LustreError: 234857:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9466.375218] LustreError: 234857:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9468.579673] LustreError: 234843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589791 with bad export cookie 13874386052522609317 [ 9468.605841] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9469.249162] Lustre: server umount lustre-MDT0001 complete [ 9483.427399] Lustre: server umount lustre-OST0000 complete [ 9487.328095] Lustre: 228157:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788589794/real 1788589794] req@ffff9b58042ac700 x1875472002697600/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788589810 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9487.366473] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9487.384436] Lustre: Skipped 1 previous similar message [ 9487.652361] Lustre: server umount lustre-OST0001 complete [ 9495.905967] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 9512.748720] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 9552.808573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9553.026845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9553.051523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9553.117792] Lustre: lustre-MDT0000: new disk, initializing [ 9553.195995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9553.209879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9558.944931] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9570.205762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9570.355923] Lustre: 241136:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9570.394797] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9570.398124] Lustre: Skipped 1 previous similar message [ 9570.447355] Lustre: lustre-MDT0001: new disk, initializing [ 9570.557653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9570.573136] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9576.016770] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9588.215374] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9597.070785] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9606.865943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9607.315256] Lustre: lustre-OST0000: new disk, initializing [ 9607.321162] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9607.331290] Lustre: 243108:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9609.166203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9609.181813] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9609.340083] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9616.106213] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9625.672828] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9630.020940] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9630.219554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9636.116587] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9636.412219] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9639.905472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9639.907668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9639.917893] LustreError: Skipped 1 previous similar message [ 9639.918777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9639.952673] Lustre: Skipped 1 previous similar message [ 9645.510207] Lustre: server umount lustre-OST0000 complete [ 9652.705732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9652.733251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9656.867724] Lustre: server umount lustre-MDT0000 complete [ 9660.394875] LustreError: 241147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9660.425383] LustreError: 241147:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9661.944876] LustreError: 241128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589984 with bad export cookie 13874386052522611452 [ 9661.952734] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9661.972035] LustreError: 241128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9662.683742] Lustre: server umount lustre-MDT0001 complete [ 9672.397495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9672.701467] LustreError: 245326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9672.747668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9672.753731] Lustre: Skipped 2 previous similar messages [ 9677.717391] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9688.043973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9694.717887] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9706.885992] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9715.351636] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9722.854409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9729.145299] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9737.858771] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9886.179562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9886.180184] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9886.182935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9886.182944] Lustre: Skipped 6 previous similar messages [ 9886.185687] Lustre: Skipped 2 previous similar messages [ 9890.383352] Lustre: server umount lustre-OST0000 complete [ 9900.665349] Lustre: server umount lustre-MDT0000 complete [ 9901.536648] LustreError: 246252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9901.556228] LustreError: 246252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9904.138657] LustreError: 245305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590227 with bad export cookie 13874386052522612334 [ 9904.163290] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9904.402464] Lustre: server umount lustre-MDT0001 complete [10009.928242] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [10013.069833] Key type lgssc unregistered [10013.353949] LNet: 248938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10013.363534] LNetError: 248938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10013.379746] LNet: Removed LNI 192.168.202.107@tcp [10014.300345] Key type .llcrypt unregistered [10014.303429] Key type ._llcrypt unregistered [10044.966191] Key type ._llcrypt registered [10044.967971] Key type .llcrypt registered [10045.095709] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [10064.042672] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [10065.271050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10065.319525] alg: No test for adler32 (adler32-zlib) [10066.684621] Lustre: Lustre: Build Version: 2.17.57_103_g3d8212a [10067.082649] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [10068.865120] Key type lgssc registered [10070.485023] Lustre: Echo OBD driver; http://www.lustre.org/ [10130.133699] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [10144.104785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10144.145439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10145.403541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10145.445218] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10145.530434] Lustre: lustre-MDT0000: new disk, initializing [10145.595930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10145.614248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10151.225348] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10163.634932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10163.813758] Lustre: 253006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10163.926144] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10163.929723] Lustre: Skipped 1 previous similar message [10164.001909] Lustre: lustre-MDT0001: new disk, initializing [10164.135503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10164.186861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10164.204505] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10168.980728] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10173.938825] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10180.746711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10181.117718] Lustre: lustre-OST0000: new disk, initializing [10181.121229] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10181.129543] Lustre: 254639:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10181.243771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10185.347173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10185.355136] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10185.439814] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10187.415291] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10198.320721] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10198.507544] Lustre: lustre-OST0001: new disk, initializing [10198.510483] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10198.516317] Lustre: 255510:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10198.616959] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10205.839214] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10207.315792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10207.339960] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10207.446378] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10217.010554] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10226.203701] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10240.992920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10241.007430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10241.028588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10243.555426] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10243.567131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10243.585254] Lustre: Skipped 2 previous similar messages [10243.601399] Lustre: Skipped 3 previous similar messages [10245.719358] Lustre: server umount lustre-MDT0000 complete [10249.537544] LustreError: 253000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590572 with bad export cookie 4469492870635537839 [10249.538278] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10249.560578] LustreError: 253000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10250.052312] Lustre: server umount lustre-MDT0001 complete [10264.604716] Lustre: server umount lustre-OST0000 complete [10279.077202] Lustre: server umount lustre-OST0001 complete [10294.122542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10303.775929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10312.834497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10324.007943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10345.912722] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [10356.807127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10356.831249] Lustre: lustre-MDT0000: reset Object Index mappings [10357.345121] LustreError: 260521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10357.360097] LustreError: 260521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10357.443863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10362.158816] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10362.854279] LustreError: 260522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10367.976399] LustreError: 260521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10371.452148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10371.485282] Lustre: lustre-MDT0001: reset Object Index mappings [10371.909738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10376.748925] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10379.967913] Lustre: 261665:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10386.701971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10394.481100] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10396.328966] LustreError: 262019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10401.772169] LustreError: 262454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10402.806177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10404.295237] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10404.645512] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10404.652225] Lustre: Skipped 1 previous similar message [10409.969314] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10411.698929] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10420.506507] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10424.868463] Lustre: 263535:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10430.437530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10430.444259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10430.467627] Lustre: Skipped 2 previous similar messages [10430.476037] Lustre: Skipped 1 previous similar message [10432.468212] Lustre: server umount lustre-MDT0000 complete [10435.556824] LustreError: 260522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10435.575175] LustreError: 260522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10436.342402] LustreError: 263540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590759 with bad export cookie 4469492870635544629 [10436.346492] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10436.351271] LustreError: 263540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10436.931474] Lustre: server umount lustre-MDT0001 complete [10451.789556] Lustre: server umount lustre-OST0000 complete [10467.035821] Lustre: server umount lustre-OST0001 complete [10489.767366] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 02:46:50 (1788590810) [10506.574332] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [10518.515307] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [10555.950372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10556.255189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10556.297668] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10556.378306] Lustre: lustre-MDT0000: new disk, initializing [10556.510491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10556.537680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10561.955141] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10571.454483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10571.568537] Lustre: 267795:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10571.615598] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10571.620723] Lustre: Skipped 1 previous similar message [10571.697558] Lustre: lustre-MDT0001: new disk, initializing [10571.812315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10571.824244] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10576.468872] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10586.527924] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10592.392030] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10599.705820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10599.998328] Lustre: lustre-OST0000: new disk, initializing [10600.002442] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10600.010736] Lustre: 269765:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10601.576106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10601.594505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10601.796393] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10606.376757] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10616.322183] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10620.844549] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10621.096477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10626.297161] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10626.533907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10632.690081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10632.704416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10632.712916] Lustre: Skipped 1 previous similar message [10632.719190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10632.729279] Lustre: Skipped 3 previous similar messages [10635.076897] Lustre: server umount lustre-OST0000 complete [10642.914726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10642.917309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10642.926660] Lustre: Skipped 2 previous similar messages [10642.943145] Lustre: Skipped 1 previous similar message [10652.130275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10652.138974] Lustre: Skipped 3 previous similar messages [10653.151162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10653.329497] Lustre: server umount lustre-MDT0000 complete [10656.832907] LustreError: 267789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590979 with bad export cookie 4469492870635546715 [10656.844780] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10656.852169] LustreError: 267789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10657.157218] Lustre: server umount lustre-MDT0001 complete [10677.744744] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [10688.676678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10689.051816] LustreError: 272701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10689.108112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10689.118100] Lustre: Skipped 2 previous similar messages [10693.392537] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10702.040507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10707.356824] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10710.445968] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10717.457859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10724.340879] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10732.767086] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10732.910589] Lustre: lustre-OST0001: new disk, initializing [10732.913416] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10732.923951] Lustre: 275009:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10734.962939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10734.971523] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10735.148611] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10739.463961] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10750.965345] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10756.038528] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10762.277880] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10762.348416] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10771.940797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10771.943059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10771.955538] Lustre: Skipped 2 previous similar messages [10771.979255] Lustre: Skipped 3 previous similar messages [10778.082853] Lustre: server umount lustre-MDT0000 complete [10782.064463] LustreError: 272683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591105 with bad export cookie 4469492870635547604 [10782.066386] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10782.079681] LustreError: 272683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10782.181354] LustreError: 272698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10782.201278] LustreError: 272698:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10782.623480] Lustre: server umount lustre-MDT0001 complete [10788.812554] Lustre: server umount lustre-OST0000 complete [10793.234873] Lustre: server umount lustre-OST0001 complete [10813.138798] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [10823.284709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10823.725221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10823.730585] Lustre: Skipped 3 previous similar messages [10828.392962] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10837.712428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10842.504982] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10845.317611] Lustre: 279205:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10852.596449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10860.141870] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10862.241076] LustreError: 279557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10862.267442] LustreError: 279557:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10869.042991] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10870.331938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10870.420796] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10877.351921] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10886.972331] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10891.221524] Lustre: 281083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10897.047432] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10897.125187] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10908.684883] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 02:53:49 (1788591229) [10925.024103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10925.031793] LustreError: Skipped 1 previous similar message [10925.037556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10925.048518] Lustre: Skipped 3 previous similar messages [10925.055154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10925.066028] Lustre: Skipped 6 previous similar messages [10938.335407] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10938.723351] Lustre: server umount lustre-MDT0000 complete [10943.731965] LustreError: 278045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591266 with bad export cookie 4469492870635554359 [10943.744026] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10943.747540] LustreError: 278045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10958.305785] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10958.626952] Lustre: server umount lustre-MDT0001 complete [10970.279706] Lustre: server umount lustre-OST0000 complete [10974.346741] Lustre: server umount lustre-OST0001 complete [10981.280900] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [10993.977261] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [11049.218761] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [11062.514987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11062.895082] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11062.954871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11063.039152] Lustre: lustre-MDT0000: new disk, initializing [11063.249867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11068.940608] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11080.320316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11080.509921] Lustre: 285329:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11080.748553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11080.753456] Lustre: Skipped 4 previous similar messages [11080.803239] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11085.817887] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11098.693853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11098.952216] Lustre: 286962:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11100.593039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11105.816594] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11118.053564] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11118.415439] Lustre: 287833:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11120.402924] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11120.413018] Lustre: Skipped 1 previous similar message [11120.577122] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11126.826350] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11138.005589] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11142.785634] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11142.791845] Lustre: Skipped 1 previous similar message [11160.722533] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 02:58:01 (1788591481) [11165.310526] Lustre: server umount lustre-MDT0000 complete [11166.697702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11166.699616] LustreError: 287841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11166.706403] Lustre: Skipped 6 previous similar messages [11166.725852] LustreError: 287841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [11168.845197] LustreError: 285322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591491 with bad export cookie 4469492870635561814 [11168.851051] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11168.854690] LustreError: 285322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11169.194740] Lustre: server umount lustre-MDT0001 complete [11182.854902] Lustre: server umount lustre-OST0000 complete [11196.702392] Lustre: server umount lustre-OST0001 complete [11205.244735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11210.786776] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11219.408188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11224.847488] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11232.417192] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11238.453393] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11246.635359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11253.169652] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11261.997352] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11282.852178] loop1: detected capacity change from 0 to 204800000 [11283.043424] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11283.059097] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11283.077426] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11288.947237] loop1: detected capacity change from 0 to 204800000 [11295.528936] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11295.616745] LustreError: 290072:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs26654: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11295.636473] LustreError: 290072:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs26654 [11295.642424] LustreError: 290072:0:(mgs_handler.c:517:mgs_target_reg()) fs26654-MDT0001: Error checking target index: rc = -13 [11295.650167] LustreError: fs26654-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11295.655115] LustreError: 293800:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11295.671937] LustreError: 293800:0:(tgt_mount.c:2132:server_put_super()) no obd fs26654-MDT0001 [11295.677414] LustreError: 293800:0:(tgt_mount.c:116:server_deregister_mount()) fs26654-MDT0001 not registered [11295.705680] Lustre: server umount fs26654-MDT0001 complete [11295.708881] LustreError: 293800:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11305.557364] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11305.692817] Lustre: Setting parameter fs26654-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs26654-MDT0001 [11305.746326] Lustre: srv-fs26654-MDT0001: No data found on store. Initialize space. [11305.750713] Lustre: Skipped 4 previous similar messages [11305.867306] Lustre: fs26654-MDT0001: new disk, initializing [11305.870909] Lustre: Skipped 3 previous similar messages [11306.017252] Lustre: srv-fs26654-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11306.028986] Lustre: 294267:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs26654-MDT0001-mdtlov: can't create new llog: rc = -115 [11306.036714] LustreError: 294267:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs26654-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11310.633957] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11318.505870] Lustre: server umount fs26654-MDT0001 complete [11331.058392] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11333.903524] loop1: detected capacity change from 0 to 204800000 [11341.003898] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11341.660988] Lustre: 295847:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs26654-MDT0001-mdtlov: can't create new llog: rc = -115 [11341.674901] LustreError: 295847:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs26654-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11346.414337] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11347.146251] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11352.978219] Lustre: server umount fs26654-MDT0001 complete [11365.415169] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.7@tcp (stopping) [11365.420258] Lustre: Skipped 20 previous similar messages [11366.882310] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11366.911672] Lustre: Skipped 4 previous similar messages [11379.167167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11379.516126] Lustre: server umount lustre-MDT0000 complete [11390.857243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11391.246522] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11397.127549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11399.448575] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11409.336871] loop1: detected capacity change from 0 to 204800000 [11416.091285] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11416.572247] Lustre: 298298:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs26654-MDT0001-mdtlov: can't create new llog: rc = -115 [11416.580375] LustreError: 298298:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs26654-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11422.235228] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11426.834162] LustreError: 292535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11426.860877] LustreError: 292535:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 106 previous similar messages [11427.138825] Lustre: server umount fs26654-MDT0001 complete [11453.428892] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11457.208608] Lustre: server umount lustre-OST0000 complete [11468.001611] Lustre: server umount lustre-MDT0000 complete [11472.634798] LustreError: 290063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591795 with bad export cookie 4469492870635564495 [11472.644401] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11472.651039] LustreError: 290063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11473.392235] Lustre: server umount lustre-MDT0001 complete [11485.176490] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [11488.698118] Key type lgssc unregistered [11488.975958] LNet: 300640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11488.983607] LNetError: 300640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11490.021880] LNet: Removed LNI 192.168.202.107@tcp [11491.043159] Key type .llcrypt unregistered [11491.045897] Key type ._llcrypt unregistered [11519.202807] Key type ._llcrypt registered [11519.205712] Key type .llcrypt registered [11519.436676] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [11520.927563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11520.985730] alg: No test for adler32 (adler32-zlib) [11523.054651] Lustre: Lustre: Build Version: 2.17.57_103_g3d8212a [11523.645739] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [11525.335556] Key type lgssc registered [11527.221310] Lustre: Echo OBD driver; http://www.lustre.org/ [11535.927406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11535.946751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11537.574658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11543.264955] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11547.128704] LustreError: 301908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11547.161947] LustreError: 301908:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11552.224370] LustreError: 301909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11554.743531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11555.362742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11560.385794] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11568.987405] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11575.900924] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11583.258455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11583.760307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11590.096923] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11598.912605] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11600.115733] LustreError: 303741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11605.513542] LustreError: 303740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11611.177667] LustreError: 303742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11611.212809] LustreError: 303742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11621.402529] LustreError: 303741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11621.443514] LustreError: 303741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11631.350892] loop1: detected capacity change from 0 to 204800000 [11631.522630] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11631.534676] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11631.570105] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11638.246630] LustreError: 304410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11638.278157] LustreError: 304410:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11638.455716] loop1: detected capacity change from 0 to 204800000 [11646.128804] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11646.382081] Lustre: fs4259-OST0000: new disk, initializing [11646.394281] Lustre: srv-fs4259-OST0000: No data found on store. Initialize space. [11646.404978] Lustre: 305671:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11646.535586] Lustre: fs4259-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11653.786038] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11662.860836] Lustre: server umount fs4259-OST0000 complete [11672.586725] LustreError: 303741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11672.623459] LustreError: 303741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [11675.543624] loop1: detected capacity change from 0 to 204800000 [11679.213820] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11681.611135] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11681.841907] Lustre: fs4259-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11684.332560] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11688.554262] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11698.295592] loop2: detected capacity change from 0 to 204800000 [11698.621508] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11698.638844] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11698.654097] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11704.528735] loop2: detected capacity change from 0 to 204800000 [11712.104426] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11712.282624] LustreError: fs4259-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11712.304404] LustreError: fs4259-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11712.314825] LustreError: 308713:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11712.320448] LustreError: 308713:0:(tgt_mount.c:2132:server_put_super()) no obd fs4259-OST0001 [11712.322497] LustreError: 308713:0:(tgt_mount.c:116:server_deregister_mount()) fs4259-OST0001 not registered [11712.362598] Lustre: server umount fs4259-OST0001 complete [11712.366484] LustreError: 308713:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11727.793962] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11728.113473] Lustre: fs4259-OST0001: new disk, initializing [11728.122943] Lustre: srv-fs4259-OST0001: No data found on store. Initialize space. [11728.145127] Lustre: 309265:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11728.311770] Lustre: fs4259-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11735.960169] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11739.152108] LustreError: 303740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11739.171924] LustreError: 303740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11745.640954] Lustre: server umount fs4259-OST0000 complete [11756.292780] Lustre: server umount fs4259-OST0001 complete [11770.353745] loop1: detected capacity change from 0 to 204800000 [11777.142161] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11777.470521] Lustre: fs4259-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11785.523867] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11793.473281] loop2: detected capacity change from 0 to 204800000 [11799.640579] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11806.237959] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11816.066655] Lustre: server umount fs4259-OST0000 complete [11825.549285] Lustre: server umount fs4259-OST0001 complete [11832.816886] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11837.943083] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11841.000328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11841.012649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11843.040717] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11843.048850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11843.057278] LustreError: Skipped 1 previous similar message [11843.057337] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11843.079499] Lustre: Skipped 1 previous similar message [11848.160770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11848.177443] Lustre: Skipped 1 previous similar message [11853.279279] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11853.293604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11853.304199] Lustre: Skipped 1 previous similar message [11853.484041] Lustre: server umount lustre-OST0000 complete [11858.401144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11858.402491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11858.414169] Lustre: Skipped 1 previous similar message [11858.432484] Lustre: Skipped 1 previous similar message [11868.647522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11868.659633] Lustre: Skipped 3 previous similar messages [11872.224929] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11872.543595] Lustre: server umount lustre-MDT0000 complete [11873.760822] LustreError: 301905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11873.796538] LustreError: 301905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 65 previous similar messages [11876.491468] LustreError: 302653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592199 with bad export cookie 8716186832529152080 [11876.496861] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11876.513216] LustreError: 302653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11876.902269] Lustre: server umount lustre-MDT0001 complete [11889.237415] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [11891.491946] Key type lgssc unregistered [11891.746777] LNet: 315192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11891.755394] LNetError: 315192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11891.770184] LNet: Removed LNI 192.168.202.107@tcp [11892.498196] Key type .llcrypt unregistered [11892.500224] Key type ._llcrypt unregistered [11927.182489] Key type ._llcrypt registered [11927.185236] Key type .llcrypt registered [11927.340490] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [11943.922500] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [11944.848628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11944.897177] alg: No test for adler32 (adler32-zlib) [11945.990788] Lustre: Lustre: Build Version: 2.17.57_103_g3d8212a [11946.204766] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [11947.856608] Key type lgssc registered [11949.226058] Lustre: Echo OBD driver; http://www.lustre.org/ [11991.480365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11991.512711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11992.869903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11992.931488] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11993.031768] Lustre: lustre-MDT0000: new disk, initializing [11993.092397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11993.108506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11998.208689] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12010.355922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12010.562062] Lustre: 319048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12010.635590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12010.639765] Lustre: Skipped 1 previous similar message [12010.737484] Lustre: lustre-MDT0001: new disk, initializing [12010.846380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12010.921173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12010.941713] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12016.205710] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12025.693983] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12031.681703] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12038.567630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12038.787173] Lustre: lustre-OST0000: new disk, initializing [12038.791074] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12038.799851] Lustre: 321017:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12038.914788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12045.966964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12045.987944] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12046.163678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12046.918941] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12057.829927] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12060.265619] LustreError: 317221:0:(lmv_obd.c:211:lmv_notify()) activation of lustre-MDT0000_UUID failed: -22 [12060.321445] Lustre: Mounted lustre-client - version 2.17.57_103_g3d8212a [12061.475676] Lustre: Unmounted lustre-client [12071.904755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12071.907582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12071.919550] LustreError: Skipped 1 previous similar message [12071.925909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12071.957162] Lustre: Skipped 1 previous similar message [12076.014554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12076.022313] Lustre: Skipped 2 previous similar messages [12077.910432] Lustre: server umount lustre-OST0000 complete [12082.657803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12082.677437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12082.724848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12086.241289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12086.241944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12086.275080] Lustre: Skipped 1 previous similar message [12090.688283] Lustre: server umount lustre-MDT0000 complete [12094.972883] LustreError: 319040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592417 with bad export cookie 4493696469899535941 [12094.981102] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12094.994278] LustreError: 319040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12095.395704] Lustre: server umount lustre-MDT0001 complete [12107.911365] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [12110.524641] Key type lgssc unregistered [12115.231171] LNet: 1 local NIs in recovery (showing 1): 192.168.202.107@tcp [12115.938218] LNet: 323179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12115.966290] LNetError: 323179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12190.757848] LNet: Removed LNI 192.168.202.107@tcp [12191.578916] Key type .llcrypt unregistered [12191.580991] Key type ._llcrypt unregistered [12208.109401] Key type ._llcrypt registered [12208.111299] Key type .llcrypt registered [12208.188535] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [12224.649897] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [12226.018845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12226.055212] alg: No test for adler32 (adler32-zlib) [12227.073989] Lustre: Lustre: Build Version: 2.17.57_103_g3d8212a [12227.290602] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [12228.999652] Key type lgssc registered [12230.240457] Lustre: Echo OBD driver; http://www.lustre.org/ [12262.464681] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 03:16:23 (1788592583) [12268.940072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12268.973662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12270.283763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12270.317451] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12270.385413] Lustre: lustre-MDT0000: new disk, initializing [12270.449330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12270.466073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12275.248494] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12285.641731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12285.816469] Lustre: 326397:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12285.868635] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12285.872853] Lustre: Skipped 1 previous similar message [12285.946540] Lustre: lustre-MDT0001: new disk, initializing [12286.054851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12286.111412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12286.150383] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12290.602894] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12300.919867] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12307.484400] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12314.824595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12315.359837] Lustre: lustre-OST0000: new disk, initializing [12315.363324] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12315.370370] Lustre: 328369:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12315.497078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12321.384676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12321.399809] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12321.555357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12322.897257] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12333.878755] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12352.482138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12352.482474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12352.496258] LustreError: Skipped 1 previous similar message [12352.512579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12352.513362] Lustre: Skipped 1 previous similar message [12352.530604] Lustre: Skipped 1 previous similar message [12354.398724] Lustre: server umount lustre-OST0000 complete [12362.720803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12362.746424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12362.750887] Lustre: Skipped 1 previous similar message [12364.336546] Lustre: server umount lustre-MDT0000 complete [12368.242369] LustreError: 326390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592691 with bad export cookie 4998092475820162877 [12368.251748] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12368.667409] Lustre: server umount lustre-MDT0001 complete [12381.675402] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [12384.831609] Key type lgssc unregistered [12385.122201] LNet: 330278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12385.133174] LNetError: 330278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12385.146602] LNet: Removed LNI 192.168.202.107@tcp [12386.019714] Key type .llcrypt unregistered [12386.022632] Key type ._llcrypt unregistered [12408.870895] Key type ._llcrypt registered [12408.876986] Key type .llcrypt registered [12408.982907] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [12410.270176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12410.324325] alg: No test for adler32 (adler32-zlib) [12411.618288] Lustre: Lustre: Build Version: 2.17.57_103_g3d8212a [12411.955123] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [12413.663189] Key type lgssc registered [12415.049428] Lustre: Echo OBD driver; http://www.lustre.org/ [12425.354609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12425.394486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12426.793887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12431.438433] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12436.971959] LustreError: 331548:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12436.998346] LustreError: 331548:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12441.486970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12441.957813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12447.837892] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12455.973242] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12461.995506] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12469.349577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12469.735505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12476.139787] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12483.540737] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12495.930864] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12497.696584] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12503.520340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12503.527069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12503.549944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12505.628905] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.7@tcp (stopping) [12506.607399] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12508.026688] Lustre: server umount lustre-MDT0000 complete [12510.728162] LustreError: 332487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12512.001179] LustreError: 331528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592834 with bad export cookie 13874310908925564704 [12512.015462] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12515.843602] LustreError: 331543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12515.866823] LustreError: 331543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12515.868304] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.7@tcp (stopping) [12515.895659] Lustre: Skipped 3 previous similar messages [12516.836698] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12516.848965] Lustre: Skipped 1 previous similar message [12518.389068] Lustre: server umount lustre-MDT0001 complete [12522.039821] Lustre: server umount lustre-OST0000 complete [12530.766796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12535.111273] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12537.357177] Lustre: 335729:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.107@tcp [12539.451898] Lustre: server umount MGS complete [12548.288760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12548.830430] LustreError: 336300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12548.856464] LustreError: 336300:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12548.904888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12553.033940] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12554.208412] LustreError: 336301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12561.801283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12562.433672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12567.847981] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12578.431051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12578.831751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12585.693263] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12594.517436] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12596.601414] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12602.653721] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12604.345558] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12610.028950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12610.047852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12610.089487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12610.107581] Lustre: Skipped 1 previous similar message [12611.545505] Lustre: server umount lustre-OST0000 complete [12618.720537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12618.733606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12618.772355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12622.447949] Lustre: server umount lustre-MDT0000 complete [12624.367658] LustreError: 336301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12624.391421] LustreError: 336301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12626.585640] LustreError: 336282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592949 with bad export cookie 13874310908925565544 [12626.602678] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12626.608231] LustreError: 336282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12636.340205] Lustre: server umount lustre-MDT0001 complete [12648.591390] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [12651.141330] Key type lgssc unregistered [12651.375893] LNet: 339736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12651.386334] LNetError: 339736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12651.407593] LNet: Removed LNI 192.168.202.107@tcp [12652.058659] Key type .llcrypt unregistered [12652.060749] Key type ._llcrypt unregistered [12670.331949] Key type ._llcrypt registered [12670.334348] Key type .llcrypt registered [12670.473265] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [12691.913188] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [12693.521770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12693.600108] alg: No test for adler32 (adler32-zlib) [12694.716707] Lustre: Lustre: Build Version: 2.17.57_103_g3d8212a [12695.028896] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [12696.745521] Key type lgssc registered [12697.965865] Lustre: Echo OBD driver; http://www.lustre.org/ [12735.918994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12735.949925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12737.278521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12737.320292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12737.416132] Lustre: lustre-MDT0000: new disk, initializing [12737.564832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12737.580496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12742.446541] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12752.427831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12752.573384] Lustre: 342891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12752.633954] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12752.636646] Lustre: Skipped 1 previous similar message [12752.689815] Lustre: lustre-MDT0001: new disk, initializing [12752.804641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12752.848051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12752.858891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12756.835303] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12766.253227] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12772.204439] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12779.183159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12779.597131] Lustre: lustre-OST0000: new disk, initializing [12779.608146] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12779.620559] Lustre: 344857:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12779.783223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12785.827685] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12786.743244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12786.752557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12786.864321] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12796.413641] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12801.981381] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12802.215326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12806.983050] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12807.303768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12812.768966] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12812.769694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12812.791750] LustreError: Skipped 1 previous similar message [12812.792417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12812.846369] Lustre: Skipped 1 previous similar message [12815.806633] Lustre: server umount lustre-OST0000 complete [12819.428460] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12819.435674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12819.447987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12819.451654] Lustre: Skipped 1 previous similar message [12823.009632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12823.011181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12825.461511] Lustre: server umount lustre-MDT0000 complete [12829.193451] LustreError: 342884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593152 with bad export cookie 15938141500935215135 [12829.199561] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12829.202349] LustreError: 342884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12829.535194] Lustre: server umount lustre-MDT0001 complete [12838.652965] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 03:25:59 (1788593159) [12858.021305] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [12875.157446] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [12915.480935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12915.810919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12915.841503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12915.955352] Lustre: lustre-MDT0000: new disk, initializing [12916.031968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12916.049685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12920.846071] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12931.781632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12931.940901] Lustre: 349642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12931.990377] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12931.996096] Lustre: Skipped 1 previous similar message [12932.073494] Lustre: lustre-MDT0001: new disk, initializing [12932.169270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12932.211822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12932.227905] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12936.151485] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12946.514107] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12953.431428] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12961.955653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12962.244320] Lustre: lustre-OST0000: new disk, initializing [12962.247560] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12962.254058] Lustre: 351611:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12962.340941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12970.324091] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12972.690399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12972.706783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12972.826963] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12981.007984] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13019.115576] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13019.118804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13019.138878] LustreError: Skipped 1 previous similar message [13019.140677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13019.140686] Lustre: Skipped 1 previous similar message [13019.169301] Lustre: Skipped 1 previous similar message [13022.639366] Lustre: server umount lustre-OST0000 complete [13029.349356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13029.364506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13029.371140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13029.371152] Lustre: Skipped 1 previous similar message [13029.379954] Lustre: Skipped 1 previous similar message [13033.117657] Lustre: server umount lustre-MDT0000 complete [13034.465358] LustreError: 352333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13034.486362] LustreError: 352333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13036.874568] LustreError: 349632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593359 with bad export cookie 15938141500935216024 [13036.876361] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13036.896490] LustreError: 349632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13037.492058] Lustre: server umount lustre-MDT0001 complete [13052.460586] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13055.913872] Key type lgssc unregistered [13056.399524] LNet: 353786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13056.415847] LNetError: 353786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13056.452213] LNet: Removed LNI 192.168.202.107@tcp [13057.276671] Key type .llcrypt unregistered [13057.279676] Key type ._llcrypt unregistered [13166.795142] Key type ._llcrypt registered [13166.802084] Key type .llcrypt registered [13166.956084] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13167.318296] Key type .llcrypt unregistered [13167.319942] Key type ._llcrypt unregistered [13247.397241] Key type ._llcrypt registered [13247.400622] Key type .llcrypt registered [13247.501984] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13247.750532] Key type .llcrypt unregistered [13247.752787] Key type ._llcrypt unregistered [13284.447155] Key type ._llcrypt registered [13284.451659] Key type .llcrypt registered [13284.694794] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13285.294461] Key type .llcrypt unregistered [13285.296490] Key type ._llcrypt unregistered [13318.352263] Key type ._llcrypt registered [13318.354482] Key type .llcrypt registered [13318.449088] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13318.711579] Key type .llcrypt unregistered [13318.713289] Key type ._llcrypt unregistered [13407.746925] Key type ._llcrypt registered [13407.751260] Key type .llcrypt registered [13407.871352] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13408.231945] Key type .llcrypt unregistered [13408.234417] Key type ._llcrypt unregistered [13441.043604] Key type ._llcrypt registered [13441.045255] Key type .llcrypt registered [13441.154265] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13441.365626] Key type .llcrypt unregistered [13441.369954] Key type ._llcrypt unregistered [13474.090305] Key type ._llcrypt registered [13474.093121] Key type .llcrypt registered [13474.221372] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13474.529364] Key type .llcrypt unregistered [13474.531834] Key type ._llcrypt unregistered [13508.331359] Key type ._llcrypt registered [13508.339660] Key type .llcrypt registered [13508.511546] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13509.135674] Key type .llcrypt unregistered [13509.141442] Key type ._llcrypt unregistered [13544.462974] Key type ._llcrypt registered [13544.467202] Key type .llcrypt registered [13544.574870] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13544.819436] Key type .llcrypt unregistered [13544.821219] Key type ._llcrypt unregistered [13574.849734] Key type ._llcrypt registered [13574.851689] Key type .llcrypt registered [13574.931974] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13575.144945] Key type .llcrypt unregistered [13575.154849] Key type ._llcrypt unregistered [13644.776881] Key type ._llcrypt registered [13644.779295] Key type .llcrypt registered [13644.881677] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13645.153506] Key type .llcrypt unregistered [13645.155214] Key type ._llcrypt unregistered [13682.663568] Key type ._llcrypt registered [13682.666167] Key type .llcrypt registered [13682.798035] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13683.255326] Key type .llcrypt unregistered [13683.258180] Key type ._llcrypt unregistered [13723.570811] Key type ._llcrypt registered [13723.572227] Key type .llcrypt registered [13723.660432] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [13723.951343] Key type .llcrypt unregistered [13723.952587] Key type ._llcrypt unregistered [13787.457669] Key type ._llcrypt registered [13787.459798] Key type .llcrypt registered [13787.565418] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [13805.912308] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [13807.431908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13807.772119] alg: No test for adler32 (adler32-zlib) [13809.039223] Lustre: Lustre: Build Version: 2.17.57_103_g3d8212a [13809.367857] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [13811.047188] Key type lgssc registered [13812.749328] Lustre: Echo OBD driver; http://www.lustre.org/ [13853.808173] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13853.840437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13855.177610] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13855.206382] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13855.308120] Lustre: tst250fs-MDT0000: new disk, initializing [13855.449198] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13855.486516] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13860.339082] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [13871.163706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13871.276393] Lustre: 374099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13871.332210] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13871.336106] Lustre: Skipped 1 previous similar message [13871.428409] Lustre: tst250fs-MDT0001: new disk, initializing [13871.533565] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13871.582380] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13871.599513] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13876.288962] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [13880.804644] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13887.993773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13888.340293] Lustre: tst250fs-OST0000: new disk, initializing [13888.345674] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13888.357696] Lustre: 375731:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13888.466695] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13890.512562] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13890.528158] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13890.707762] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13895.104606] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [13907.029372] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13907.250976] Lustre: tst250fs-OST0001: new disk, initializing [13907.255355] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13907.269385] Lustre: 376601:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13907.455218] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13912.777511] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13912.798653] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13912.949461] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13914.609757] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [13921.704996] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13924.748040] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13924.760661] Lustre: Skipped 1 previous similar message [13948.384672] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13948.391816] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13948.414092] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13948.922036] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13948.942609] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13950.317727] LustreError: 377923:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff9b5809af0400) refcount nonzero (2) after lock cleanup; forcing cleanup. [13950.571736] Lustre: server umount tst250fs-MDT0000 complete [13954.605070] LustreError: 374093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594277 with bad export cookie 11647489976198021977 [13954.609413] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13954.623915] LustreError: 374093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13955.246750] Lustre: server umount tst250fs-MDT0001 complete [13969.878311] Lustre: server umount tst250fs-OST0000 complete [13983.302603] Lustre: server umount tst250fs-OST0001 complete [14003.686535] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [14015.442836] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [14059.104610] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 03:46:19 (1788594379) [14061.224566] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14082.396275] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [14096.545614] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [14132.078925] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13805 sec ========= 03:47:32 (1788594452) [14133.965607] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:47:34 (1788594454) === [14138.218451] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:47:38 (1788594458) === [14183.180909] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [14186.389769] Key type lgssc unregistered [14186.614333] LNet: 385766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14186.626609] LNetError: 385766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14186.649371] LNet: Removed LNI 192.168.202.107@tcp [14187.451234] Key type .llcrypt unregistered [14187.454118] Key type ._llcrypt unregistered