[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 485584219 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003353] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010016] pid_max: default: 32768 minimum: 301 [ 0.012134] LSM: Security Framework initializing [ 0.013056] Yama: becoming mindful. [ 0.014044] SELinux: Initializing. [ 0.015065] *** VALIDATE selinux *** [ 0.023795] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027588] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028158] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029106] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030108] *** VALIDATE tmpfs *** [ 0.031470] *** VALIDATE proc *** [ 0.033135] *** VALIDATE cgroup *** [ 0.034013] *** VALIDATE cgroup2 *** [ 0.035271] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036145] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.041782] debug: unmapping init [mem 0xffffffffbc059000-0xffffffffbc060fff] [ 0.043177] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044714] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045024] ... version: 2 [ 0.046013] ... bit width: 48 [ 0.047012] ... generic registers: 4 [ 0.048013] ... value mask: 0000ffffffffffff [ 0.049013] ... max period: 00007fffffffffff [ 0.050012] ... fixed-purpose events: 3 [ 0.051011] ... event mask: 000000070000000f [ 0.052257] rcu: Hierarchical SRCU implementation. [ 0.054278] smp: Bringing up secondary CPUs ... [ 0.055519] x86: Booting SMP configuration: [ 0.056037] .... node #0, CPUs: #1 #2 #3 [ 0.059742] smp: Brought up 1 node, 4 CPUs [ 0.061013] smpboot: Max logical packages: 1 [ 0.062020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.230821] node 0 deferred pages initialised in 165ms [ 0.234179] devtmpfs: initialized [ 0.236281] x86/mm: Memory block size: 128MB [ 0.240211] gcov: version magic: 0x41383552 [ 0.243484] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.249140] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.252591] pinctrl core: initialized pinctrl subsystem [ 0.255356] [ 0.256000] ************************************************************* [ 0.259020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.262018] ** ** [ 0.265022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.268015] ** ** [ 0.270016] ** This means that this kernel is built to expose internal ** [ 0.271016] ** IOMMU data structures, which may compromise security on ** [ 0.274020] ** your system. ** [ 0.277019] ** ** [ 0.280018] ** If you see this message and you are not debugging the ** [ 0.282016] ** kernel, report this immediately to your vendor! ** [ 0.285021] ** ** [ 0.289020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.292017] ************************************************************* [ 0.294874] NET: Registered protocol family 16 [ 0.297539] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.302112] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.305176] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.309136] cpuidle: using governor menu [ 0.310641] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.311447] PCI: Using configuration type 1 for base access [ 0.312120] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.319154] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.320027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.322161] cryptd: max_cpu_qlen set to 1000 [ 0.326160] ACPI: Added _OSI(Module Device) [ 0.328013] ACPI: Added _OSI(Processor Device) [ 0.330012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.331013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.336252] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.343430] ACPI: Interpreter enabled [ 0.345085] ACPI: PM: (supports S0 S3 S4 S5) [ 0.347043] ACPI: Using IOAPIC for interrupt routing [ 0.349186] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.353485] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.364260] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.367070] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.371027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.376099] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.382514] acpiphp: Slot [2] registered [ 0.384195] acpiphp: Slot [5] registered [ 0.385000] acpiphp: Slot [6] registered [ 0.385000] acpiphp: Slot [7] registered [ 0.388187] acpiphp: Slot [8] registered [ 0.390242] acpiphp: Slot [9] registered [ 0.392214] acpiphp: Slot [10] registered [ 0.394211] acpiphp: Slot [3] registered [ 0.396154] acpiphp: Slot [4] registered [ 0.398192] acpiphp: Slot [11] registered [ 0.400163] acpiphp: Slot [12] registered [ 0.401164] acpiphp: Slot [13] registered [ 0.403179] acpiphp: Slot [14] registered [ 0.405159] acpiphp: Slot [15] registered [ 0.407180] acpiphp: Slot [16] registered [ 0.409136] acpiphp: Slot [17] registered [ 0.411136] acpiphp: Slot [18] registered [ 0.412158] acpiphp: Slot [19] registered [ 0.414185] acpiphp: Slot [20] registered [ 0.416155] acpiphp: Slot [21] registered [ 0.418141] acpiphp: Slot [22] registered [ 0.419139] acpiphp: Slot [23] registered [ 0.421160] acpiphp: Slot [24] registered [ 0.423155] acpiphp: Slot [25] registered [ 0.425161] acpiphp: Slot [26] registered [ 0.427273] acpiphp: Slot [27] registered [ 0.428124] acpiphp: Slot [28] registered [ 0.430153] acpiphp: Slot [29] registered [ 0.432167] acpiphp: Slot [30] registered [ 0.434145] acpiphp: Slot [31] registered [ 0.436095] PCI host bridge to bus 0000:00 [ 0.437025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.440043] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.443049] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.447062] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.450033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.454045] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.457300] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.460145] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.463555] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.476020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.482066] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.487028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.490029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.493070] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.496125] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.501517] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.504046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.508940] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.514014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.529016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.534996] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.543721] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.553023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.561024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.582021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.592327] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.606014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.615016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.637021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.649730] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.658026] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.669025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.685025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.695539] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.707016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.717023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.737030] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.749298] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.759021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.766025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.788027] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.801634] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.813017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.818015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.843022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.857354] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.860477] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.862399] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.865518] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.868306] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.873143] iommu: Default domain type: Passthrough [ 0.875516] SCSI subsystem initialized [ 0.877166] ACPI: bus type USB registered [ 0.879131] usbcore: registered new interface driver usbfs [ 0.881101] usbcore: registered new interface driver hub [ 0.882129] usbcore: registered new device driver usb [ 0.884207] pps_core: LinuxPPS API ver. 1 registered [ 0.886016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.889109] PTP clock support registered [ 0.891160] EDAC MC: Ver: 3.0.0 [ 0.893160] PCI: Using ACPI for IRQ routing [ 0.895873] NetLabel: Initializing [ 0.897014] NetLabel: domain hash size = 128 [ 0.898015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.900116] NetLabel: unlabeled traffic allowed by default [ 0.902193] vgaarb: loaded [ 0.904315] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.906015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.913012] clocksource: Switched to clocksource kvm-clock [ 1.036618] VFS: Disk quotas dquot_6.6.0 [ 1.039257] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.041537] *** VALIDATE ramfs *** [ 1.042731] *** VALIDATE hugetlbfs *** [ 1.044055] pnp: PnP ACPI init [ 1.046767] pnp: PnP ACPI: found 6 devices [ 1.066127] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.070056] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.072701] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.075386] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.078027] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.080892] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.084108] NET: Registered protocol family 2 [ 1.086993] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.093674] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.098001] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.103677] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.106925] TCP: Hash tables configured (established 65536 bind 65536) [ 1.109925] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.113234] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.116398] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.119867] NET: Registered protocol family 1 [ 1.123610] RPC: Registered named UNIX socket transport module. [ 1.126208] RPC: Registered udp transport module. [ 1.128261] RPC: Registered tcp transport module. [ 1.130325] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.132294] NET: Registered protocol family 44 [ 1.134202] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.136524] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.138807] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.141591] PCI: CLS 0 bytes, default 64 [ 1.143410] Unpacking initramfs... [ 2.558652] debug: unmapping init [mem 0xffff990dfcc54000-0xffff990dfffbffff] [ 2.563167] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.565738] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.568887] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.058222] Initialise system trusted keyrings [ 3.060223] Key type blacklist registered [ 3.062752] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.072172] zbud: loaded [ 3.075545] *** VALIDATE nfs *** [ 3.077192] *** VALIDATE nfs4 *** [ 3.079167] pstore: using deflate compression [ 3.084489] Platform Keyring initialized [ 3.185555] NET: Registered protocol family 38 [ 3.187305] Key type asymmetric registered [ 3.188362] Asymmetric key parser 'x509' registered [ 3.189975] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.193102] io scheduler mq-deadline registered [ 3.194572] io scheduler kyber registered [ 3.196183] io scheduler bfq registered [ 3.199087] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.201749] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.205405] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.208981] ACPI: Power Button [PWRF] [ 3.215061] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.222580] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.235358] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.245357] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.261441] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.291601] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.323156] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.329508] Non-volatile memory driver v1.3 [ 3.332705] Linux agpgart interface v0.103 [ 3.365078] virtio_blk virtio1: [vda] 149816 512-byte logical blocks (76.7 MB/73.2 MiB) [ 3.368599] vda: detected capacity change from 0 to 76705792 [ 3.383737] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.387359] vdb: detected capacity change from 0 to 1073741824 [ 3.403786] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.407459] vdc: detected capacity change from 0 to 2621440000 [ 3.425652] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.428963] vdd: detected capacity change from 0 to 2621440000 [ 3.443280] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.446830] vde: detected capacity change from 0 to 4294967296 [ 3.463559] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.466361] vdf: detected capacity change from 0 to 4294967296 [ 3.471993] libphy: Fixed MDIO Bus: probed [ 3.477538] usbcore: registered new interface driver usbserial_generic [ 3.480509] usbserial: USB Serial support registered for generic [ 3.482931] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.487636] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.489759] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.492540] mousedev: PS/2 mouse device common for all mice [ 3.496423] rtc_cmos 00:05: RTC can wake from S4 [ 3.499932] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.500050] rtc_cmos 00:05: registered as rtc0 [ 3.507060] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.507073] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.512626] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.514328] intel_pstate: CPU model not supported [ 3.521640] hid: raw HID events driver (C) Jiri Kosina [ 3.524058] usbcore: registered new interface driver usbhid [ 3.526296] usbhid: USB HID core driver [ 3.528109] drop_monitor: Initializing network drop monitor service [ 3.530772] Initializing XFRM netlink socket [ 3.533174] NET: Registered protocol family 10 [ 3.537312] Segment Routing with IPv6 [ 3.539044] NET: Registered protocol family 17 [ 3.541569] mpls_gso: MPLS GSO support [ 3.547583] RAS: Correctable Errors collector initialized. [ 3.550112] AVX version of gcm_enc/dec engaged. [ 3.552066] AES CTR mode by8 optimization enabled [ 3.642887] sched_clock: Marking stable (3642848599, 0)->(4551523876, -908675277) [ 3.647235] registered taskstats version 1 [ 3.651079] Loading compiled-in X.509 certificates [ 3.653351] zswap: loaded using pool lzo/zbud [ 3.682037] Key type big_key registered [ 3.694775] Key type encrypted registered [ 3.696715] ima: No TPM chip found, activating TPM-bypass! [ 3.699299] ima: Allocated hash algorithm: sha1 [ 3.701562] ima: No architecture policies found [ 3.703910] evm: Initialising EVM extended attributes: [ 3.706223] evm: security.selinux [ 3.707704] evm: security.ima [ 3.709166] evm: security.capability [ 3.710714] evm: HMAC attrs: 0x1 [ 3.713487] rtc_cmos 00:05: setting system clock to 2026-09-05 07:32:41 UTC (1788593561) [ 3.720574] debug: unmapping init [mem 0xffffffffbd003000-0xffffffffbd1fffff] [ 3.724069] debug: unmapping init [mem 0xffffffffbbd82000-0xffffffffbc058fff] [ 3.732307] Write protecting the kernel read-only data: 28672k [ 3.736428] debug: unmapping init [mem 0xffffffffba403000-0xffffffffba5fffff] [ 3.739520] debug: unmapping init [mem 0xffffffffbad14000-0xffffffffbadfffff] [ 3.781628] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.791871] systemd[1]: Detected virtualization kvm. [ 3.793575] systemd[1]: Detected architecture x86-64. [ 3.795323] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.822097] systemd[1]: No hostname configured. [ 3.825248] systemd[1]: Set hostname to . [ 3.827437] random: systemd: uninitialized urandom read (16 bytes read) [ 3.830051] systemd[1]: Initializing machine ID from random generator. [ 3.873635] random: ln: uninitialized urandom read (6 bytes read) [ 3.966833] random: systemd: uninitialized urandom read (16 bytes read) [ 3.970062] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.975228] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.984296] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.587764] device-mapper: uevent: version 1.0.3 [ 4.590155] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.348837] random: fast init done [ 5.373532] virtio_net virtio0 ens2: renamed from eth0 [ 5.431978] scsi host0: ata_piix [ 5.441311] scsi host1: ata_piix [ 5.443530] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.446244] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.767586] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.216664] random: crng init done [ 10.218206] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.649597] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.883071] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.138785] SELinux: Disabled at runtime. [ 12.202547] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.211761] systemd[1]: Detected virtualization kvm. [ 12.214042] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.733390] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.737329] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.743260] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.748253] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.752333] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.761138] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.765736] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ 12.829674] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.243239] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.573160] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.598844] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.772803] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.801128] EDAC sbridge: Ver: 1.1.2 [ 15.402431] Key type dns_resolver registered [ 15.772130] NFS: Registering the id_resolver key type [ 15.775274] Key type id_resolver registered [ 15.777548] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg314-server login: [ 57.399862] hrtimer: interrupt took 10014979 ns [ 65.663928] libcfs: loading out-of-tree module taints kernel. [ 65.737491] Key type ._llcrypt registered [ 65.739815] Key type .llcrypt registered [ 65.864498] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 83.241836] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 84.610677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 84.632729] alg: No test for adler32 (adler32-zlib) [ 86.365283] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 87.498696] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 89.247163] Key type lgssc registered [ 90.735050] Lustre: Echo OBD driver; http://www.lustre.org/ [ 107.467255] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 141.896172] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 153.612211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 153.641268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 154.903892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 154.937275] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 155.105402] Lustre: lustre-MDT0000: new disk, initializing [ 155.198729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 155.229970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 159.722369] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.026837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 173.130159] Lustre: 6507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 173.187918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 173.196584] Lustre: Skipped 1 previous similar message [ 173.288966] Lustre: lustre-MDT0001: new disk, initializing [ 173.363885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 173.409849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 173.431326] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 177.916232] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.236997] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 194.174285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 194.482346] Lustre: lustre-OST0000: new disk, initializing [ 194.486322] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 194.492799] Lustre: 8445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 194.573617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 200.260384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 200.276621] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 200.387332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 201.609945] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.808492] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 214.947590] Lustre: lustre-OST0001: new disk, initializing [ 214.950853] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 214.957761] Lustre: 9516:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 215.014181] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 221.497707] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.854269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 224.881726] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 224.949519] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 233.128090] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 238.429391] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 245.593776] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing check_logdir /tmp/testlogs/ [ 251.051574] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing yml_node [ 255.765537] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 258.311388] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 261.056539] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 262.909792] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 03:36:58 EDT 2026 [ 279.764611] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 281.119837] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 296.931653] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.937990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 296.955576] Lustre: Skipped 2 previous similar messages [ 296.967182] Lustre: Skipped 2 previous similar messages [ 301.536814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 301.543130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.573347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 302.926759] Lustre: server umount lustre-MDT0000 complete [ 306.554391] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593864 with bad export cookie 3276024456962510371 [ 306.561607] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 306.578103] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 307.168360] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.169245] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 307.177754] Lustre: Skipped 1 previous similar message [ 307.194666] Lustre: Skipped 5 previous similar messages [ 311.201478] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 311.207368] Lustre: Skipped 1 previous similar message [ 313.102113] Lustre: server umount lustre-MDT0001 complete [ 317.933084] Lustre: server umount lustre-OST0000 complete [ 322.589726] Lustre: server umount lustre-OST0001 complete [ 329.674939] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 339.133328] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 374.992141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 375.382589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 375.448174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 375.497597] Lustre: lustre-MDT0000: new disk, initializing [ 375.623073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 375.658423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 380.834266] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 391.493685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 391.569983] Lustre: 15070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 391.628742] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 391.631245] Lustre: Skipped 1 previous similar message [ 391.700048] Lustre: lustre-MDT0001: new disk, initializing [ 391.789915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 391.847633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 391.864795] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 395.517222] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 405.543333] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 410.860894] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 416.860645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 417.125370] Lustre: lustre-OST0000: new disk, initializing [ 417.129241] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 417.137904] Lustre: 17038:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 417.233838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 418.234369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 418.253454] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 418.404760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 423.194165] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 432.095811] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 436.966638] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 437.243594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 441.786060] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 442.058533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 448.995219] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 448.995948] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 449.017058] LustreError: Skipped 1 previous similar message [ 449.020309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 449.036377] Lustre: Skipped 1 previous similar message [ 449.071302] Lustre: Skipped 1 previous similar message [ 450.506695] Lustre: server umount lustre-OST0000 complete [ 458.211700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 458.223107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 458.251902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 460.813865] Lustre: server umount lustre-MDT0000 complete [ 464.365628] LustreError: 15081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 464.386150] LustreError: 15081:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 464.420167] LustreError: 15866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594022 with bad export cookie 3276024456962511904 [ 464.423840] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 464.429780] LustreError: 15866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 464.775530] Lustre: server umount lustre-MDT0001 complete [ 470.683555] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 03:40:26 (1788594026) [ 488.632803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.086030] LustreError: 19913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 489.172835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 493.535496] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 494.576310] LustreError: 19914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.664309] LustreError: 19913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 502.422310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 507.765499] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 514.502618] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 520.154139] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 522.291598] Lustre: Failing over lustre-MDT0000 [ 522.329038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.330174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 522.347734] Lustre: Skipped 2 previous similar messages [ 522.356825] Lustre: Skipped 3 previous similar messages [ 522.514791] Lustre: server umount lustre-MDT0000 complete [ 527.334325] LustreError: 19914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 527.359256] LustreError: 19914:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 538.594670] LustreError: 21154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.640666] LustreError: 21154:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 542.609162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 542.741655] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 548.234972] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 548.332638] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 548.334119] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 548.373614] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 558.310829] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 562.662352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 562.672529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 562.692514] Lustre: Skipped 1 previous similar message [ 562.699789] Lustre: Skipped 1 previous similar message [ 568.468836] Lustre: server umount lustre-MDT0000 complete [ 572.494355] LustreError: 19893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594130 with bad export cookie 3276024456962513318 [ 572.531052] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 573.032275] Lustre: server umount lustre-MDT0001 complete [ 581.937118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 582.464837] LustreError: 23247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 582.557765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 582.568443] Lustre: Skipped 2 previous similar messages [ 586.995820] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 596.506491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 602.252496] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 610.904099] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 617.139895] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 619.115306] Lustre: Failing over lustre-MDT0000 [ 619.390806] Lustre: server umount lustre-MDT0000 complete [ 622.243862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 622.247308] LustreError: 23244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 622.257331] Lustre: Skipped 1 previous similar message [ 622.274364] LustreError: 23244:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 638.409417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 638.572143] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 638.999946] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 643.057101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 643.060842] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 643.063825] Lustre: Skipped 1 previous similar message [ 643.090660] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 643.412880] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 653.163945] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 658.405133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 658.409677] Lustre: Skipped 3 previous similar messages [ 663.702332] Lustre: server umount lustre-MDT0000 complete [ 667.143838] LustreError: 23228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594224 with bad export cookie 3276024456962514256 [ 667.154734] LustreError: 23228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 667.155267] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 667.532387] Lustre: server umount lustre-MDT0001 complete [ 676.112410] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 03:43:52 (1788594232) [ 692.192422] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 704.543692] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 743.585966] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 755.744220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 755.984511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 756.007207] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 756.092598] Lustre: lustre-MDT0000: new disk, initializing [ 756.193220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 756.199849] Lustre: Skipped 2 previous similar messages [ 756.217345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 760.327310] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 771.805861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 771.945965] Lustre: 29627:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 772.267510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 776.968457] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 787.097995] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 793.519281] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 801.002093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 801.310852] Lustre: 31596:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 802.859818] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 802.920442] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 802.983334] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 803.009497] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 808.163410] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 817.691350] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 822.965116] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 844.258261] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 844.262614] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 844.279178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 844.279185] Lustre: Skipped 2 previous similar messages [ 844.308385] Lustre: Skipped 3 previous similar messages [ 845.279849] Lustre: server umount lustre-OST0000 complete [ 848.865108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 863.199091] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 863.440554] Lustre: server umount lustre-MDT0000 complete [ 864.742902] LustreError: 29639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 864.759420] LustreError: 29639:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 867.196453] LustreError: 29618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594424 with bad export cookie 3276024456962514662 [ 867.214877] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 867.789852] Lustre: server umount lustre-MDT0001 complete [ 878.663531] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 881.610186] Key type lgssc unregistered [ 882.001636] LNet: 33572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.011364] LNetError: 33572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.029368] LNet: Removed LNI 192.168.203.114@tcp [ 882.917452] Key type .llcrypt unregistered [ 882.920631] Key type ._llcrypt unregistered [ 907.591870] Key type ._llcrypt registered [ 907.594395] Key type .llcrypt registered [ 907.768245] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 909.031451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 909.152453] alg: No test for adler32 (adler32-zlib) [ 910.276567] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 910.640644] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 912.319308] Key type lgssc registered [ 913.619111] Lustre: Echo OBD driver; http://www.lustre.org/ [ 921.509864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 921.530086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 922.898906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 927.105902] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 932.334916] LustreError: 34984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 932.350879] LustreError: 34984:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 934.912038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 935.228299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 939.599315] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 946.926956] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 952.153815] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 958.971868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 959.485259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 960.512684] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 966.144516] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 967.690144] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 973.613833] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 988.044661] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 03:49:04 (1788594544) [ 989.867392] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 991.819258] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 999.228489] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 03:49:15 (1788594555) [ 1008.014362] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 03:49:24 (1788594564) [ 1009.138761] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1018.353104] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1018.363993] Lustre: Skipped 2 previous similar messages [ 1020.307155] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 03:49:36 (1788594576) [ 1024.173926] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1024.182672] Lustre: Skipped 3 previous similar messages [ 1041.094962] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1041.101265] Lustre: Skipped 6 previous similar messages [ 1043.621313] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 03:49:59 (1788594599) [ 1046.211677] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1049.964539] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1058.509283] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 03:50:14 (1788594614) [ 1060.380165] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1060.391886] Lustre: Skipped 3 previous similar messages [ 1072.984685] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 03:50:29 (1788594629) [ 1085.412796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1085.413467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1085.424213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1085.431082] Lustre: Skipped 1 previous similar message [ 1085.449377] Lustre: Skipped 1 previous similar message [ 1090.530464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1090.536630] Lustre: Skipped 1 previous similar message [ 1095.648554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1095.661888] Lustre: Skipped 1 previous similar message [ 1099.231584] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1099.564928] Lustre: server umount lustre-OST0000 complete [ 1104.352590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1104.364488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1104.390893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1105.889827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1111.013022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1111.017885] Lustre: Skipped 3 previous similar messages [ 1118.175371] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1118.507360] Lustre: server umount lustre-MDT0000 complete [ 1121.248760] LustreError: 34985:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1121.272392] LustreError: 34985:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1122.038120] LustreError: 34966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594679 with bad export cookie 2681963515175659562 [ 1122.039224] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1122.049662] LustreError: 34966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1122.585466] Lustre: server umount lustre-MDT0001 complete [ 1134.574028] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 1137.426900] Key type lgssc unregistered [ 1137.696857] LNet: 42830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1137.703942] LNetError: 42830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1137.719473] LNet: Removed LNI 192.168.203.114@tcp [ 1138.752299] Key type .llcrypt unregistered [ 1138.759407] Key type ._llcrypt unregistered [ 1157.066530] Key type ._llcrypt registered [ 1157.068694] Key type .llcrypt registered [ 1157.148345] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 1173.054927] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 1174.420331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1174.454382] alg: No test for adler32 (adler32-zlib) [ 1175.563845] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 1175.903231] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 1177.663591] Key type lgssc registered [ 1178.891020] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1212.967750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1212.984303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1214.272956] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1214.309087] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1214.390921] Lustre: lustre-MDT0000: new disk, initializing [ 1214.500237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1214.521425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1218.992079] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1229.349727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1229.480339] Lustre: 45978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1229.525520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1229.528939] Lustre: Skipped 1 previous similar message [ 1229.578873] Lustre: lustre-MDT0001: new disk, initializing [ 1229.664682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1229.700400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1229.714849] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1234.007380] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1243.270798] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1248.711544] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1255.488919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1255.749859] Lustre: lustre-OST0000: new disk, initializing [ 1255.753591] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1255.765858] Lustre: 47947:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1255.914406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1262.356904] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1265.223050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1265.243027] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1265.412447] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1270.533626] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1275.326671] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1275.593519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1280.483342] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1280.716571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1286.128727] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1286.128917] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1286.131667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1286.141789] Lustre: Skipped 1 previous similar message [ 1288.937957] Lustre: server umount lustre-OST0000 complete [ 1296.352190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1296.356809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1296.367787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1296.369902] Lustre: Skipped 1 previous similar message [ 1296.374920] Lustre: Skipped 1 previous similar message [ 1298.825949] Lustre: server umount lustre-MDT0000 complete [ 1302.167269] LustreError: 45968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594859 with bad export cookie 13689550754322385798 [ 1302.175258] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1302.185487] LustreError: 45968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1302.818187] Lustre: server umount lustre-MDT0001 complete [ 1312.056661] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 03:54:28 (1788594868) [ 1318.433031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1318.998606] LustreError: 50405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1319.020841] LustreError: 50405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1319.096855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1323.598116] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1324.516579] LustreError: 50406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1324.530334] LustreError: 50406:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1328.611417] LustreError: 50405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1331.720507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1332.092628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1335.841397] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1342.676499] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1346.881891] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1352.467616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1352.896820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1360.078934] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1368.275304] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1373.263667] LustreError: 52932:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1374.200503] Lustre: Modifying parameter general.timeout=1 in log params [ 1375.629885] Lustre: Modifying parameter general.timeout=3 in log params [ 1375.636411] Lustre: Skipped 1 previous similar message [ 1377.769462] Lustre: Modifying parameter general.timeout=6 in log params [ 1377.773840] Lustre: Skipped 2 previous similar messages [ 1382.455431] Lustre: Modifying parameter general.timeout=12 in log params [ 1382.460902] Lustre: Skipped 5 previous similar messages [ 1390.515489] Lustre: Modifying parameter general.timeout=22 in log params [ 1390.518789] Lustre: Skipped 9 previous similar messages [ 1406.603907] Lustre: Modifying parameter general.timeout=43 in log params [ 1406.616465] Lustre: Skipped 20 previous similar messages [ 1439.369145] Lustre: Modifying parameter general.timeout=81 in log params [ 1439.373640] Lustre: Skipped 37 previous similar messages [ 1493.103357] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 03:57:29 (1788595049) [ 1495.688670] Lustre: 59868:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1497.313403] Lustre: 59966:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1500.832661] Lustre: 60179:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1500.842441] Lustre: 60179:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1505.612482] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule= in log params [ 1505.623131] Lustre: Skipped 56 previous similar messages [ 1530.209289] Lustre: 61735:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1530.215436] Lustre: 61735:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1530.240034] LustreError: 61735:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1534.508488] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 03:58:10 (1788595090) [ 1547.235805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1547.254892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1547.284446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1547.297593] Lustre: Skipped 1 previous similar message [ 1553.242604] Lustre: server umount lustre-MDT0000 complete [ 1553.386890] LustreError: 50401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1553.404069] LustreError: 50401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1556.810913] LustreError: 50388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595114 with bad export cookie 13689550754322386673 [ 1556.814480] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1556.829774] LustreError: 50388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1557.218501] Lustre: server umount lustre-MDT0001 complete [ 1570.975743] Lustre: server umount lustre-OST0000 complete [ 1597.696065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1598.153446] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1598.450589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1603.187492] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1613.312664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1613.364185] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1613.380429] Lustre: 64271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1618.177102] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1626.101576] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1631.478844] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1638.545186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1638.803680] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1639.184355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1639.205919] Lustre: Skipped 1 previous similar message [ 1644.840819] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1651.129468] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1652.572517] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1652.581222] Lustre: Skipped 15 previous similar messages [ 1652.624293] Lustre: 66835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1663.969589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1663.981357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1664.001783] Lustre: Skipped 2 previous similar messages [ 1664.016588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1664.020835] Lustre: Skipped 3 previous similar messages [ 1666.152567] Lustre: server umount lustre-OST0000 complete [ 1670.122060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1670.134906] LustreError: Skipped 1 previous similar message [ 1670.140291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1670.145304] Lustre: Skipped 1 previous similar message [ 1670.150253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1670.154868] Lustre: Skipped 2 previous similar messages [ 1676.356794] Lustre: server umount lustre-MDT0000 complete [ 1679.840757] LustreError: 65306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1679.864669] LustreError: 65306:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1679.970348] LustreError: 64264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595237 with bad export cookie 13689550754322389690 [ 1679.986876] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1679.989565] LustreError: 64264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1680.311912] Lustre: server umount lustre-MDT0001 complete [ 1690.446673] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 1692.813303] Key type lgssc unregistered [ 1693.088558] LNet: 68045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1693.094289] LNetError: 68045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1693.118929] LNet: Removed LNI 192.168.203.114@tcp [ 1693.972820] Key type .llcrypt unregistered [ 1693.979142] Key type ._llcrypt unregistered [ 1718.579778] Key type ._llcrypt registered [ 1718.582656] Key type .llcrypt registered [ 1718.693573] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 1719.857601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1719.902872] alg: No test for adler32 (adler32-zlib) [ 1721.038844] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 1721.231330] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 1722.975180] Key type lgssc registered [ 1724.139849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1730.823871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1730.847494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1732.224879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1735.824292] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1741.815167] LustreError: 69468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1741.840264] LustreError: 69468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1743.326927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1743.893701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1747.492562] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1753.113987] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1758.067667] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1763.406254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1763.845166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1769.364486] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1775.636321] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1786.545745] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1795.040299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1795.051603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1795.081559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1797.600070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1797.602938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1797.612375] Lustre: Skipped 1 previous similar message [ 1797.627841] Lustre: Skipped 1 previous similar message [ 1800.835815] Lustre: server umount lustre-MDT0000 complete [ 1802.723914] LustreError: 69468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1802.759528] LustreError: 69468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1804.702342] LustreError: 69447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595362 with bad export cookie 13582038590770793642 [ 1804.707764] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1804.713386] LustreError: 69447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1804.988734] Lustre: server umount lustre-MDT0001 complete [ 1819.134204] Lustre: server umount lustre-OST0000 complete [ 1844.378655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1844.706571] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1844.751381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1844.978854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1849.170492] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1857.146925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1857.286225] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1857.343555] Lustre: 74342:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1857.897137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1862.053898] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1869.169760] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1874.470767] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1880.185608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1880.364826] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1880.662568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1886.270277] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1894.355742] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1896.023967] Lustre: 76918:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1897.803881] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1904.617153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1904.629437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1904.639448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1904.682658] LustreError: Skipped 1 previous similar message [ 1908.899751] Lustre: server umount lustre-OST0000 complete [ 1913.830736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1913.844521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1913.865499] Lustre: Skipped 1 previous similar message [ 1913.872214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1913.877021] Lustre: Skipped 1 previous similar message [ 1919.064545] Lustre: server umount lustre-MDT0000 complete [ 1919.968164] LustreError: 74353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1919.982036] LustreError: 74353:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1923.473641] LustreError: 76209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595481 with bad export cookie 13582038590770794489 [ 1923.486594] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1923.495053] LustreError: 76209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1923.926525] Lustre: server umount lustre-MDT0001 complete [ 1936.559882] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 1938.877892] Key type lgssc unregistered [ 1939.223676] LNet: 78130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1939.230912] LNetError: 78130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1939.249162] LNet: Removed LNI 192.168.203.114@tcp [ 1940.035135] Key type .llcrypt unregistered [ 1940.037243] Key type ._llcrypt unregistered [ 1963.057586] Key type ._llcrypt registered [ 1963.061314] Key type .llcrypt registered [ 1963.153498] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 1963.999193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1964.094103] alg: No test for adler32 (adler32-zlib) [ 1965.108760] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 1965.329935] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 1966.991244] Key type lgssc registered [ 1967.881973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1974.287845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1974.297954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1975.804750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1979.782783] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1980.899818] LustreError: 79540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1980.932888] LustreError: 79540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1986.021987] LustreError: 79541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1987.191025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1987.798364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1991.879992] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1998.658176] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2003.745835] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2009.486786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2010.099828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2016.666342] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2023.115285] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2028.970497] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2029.667535] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2031.131674] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2031.141895] Lustre: Skipped 1 previous similar message [ 2033.378874] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2033.384878] Lustre: Skipped 2 previous similar messages [ 2037.937535] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2037.941382] Lustre: Skipped 5 previous similar messages [ 2046.549323] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2046.553609] Lustre: Skipped 11 previous similar messages [ 2063.468383] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2063.473827] Lustre: Skipped 21 previous similar messages [ 2095.982200] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2095.990298] Lustre: Skipped 39 previous similar messages [ 2114.744794] Lustre: 87626:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2117.183596] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 04:07:53 (1788595673) [ 2118.539587] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2120.080221] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 04:07:56 (1788595676) [ 2125.798993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2125.804768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2125.823978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2127.840368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2127.844170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2127.874878] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2132.977860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2132.989949] Lustre: Skipped 2 previous similar messages [ 2138.080587] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2138.083066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2138.109845] Lustre: Skipped 1 previous similar message [ 2138.342250] Lustre: server umount lustre-OST0000 complete [ 2143.200806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2143.203741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2143.214633] Lustre: Skipped 1 previous similar message [ 2148.489954] Lustre: server umount lustre-MDT0000 complete [ 2152.138288] LustreError: 79519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595709 with bad export cookie 10201573400201558830 [ 2152.147644] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2152.152810] LustreError: 79519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2152.487410] Lustre: server umount lustre-MDT0001 complete [ 2162.222308] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 2164.686959] Key type lgssc unregistered [ 2164.914919] LNet: 88876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2164.924790] LNetError: 88876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2164.945322] LNet: Removed LNI 192.168.203.114@tcp [ 2165.590375] Key type .llcrypt unregistered [ 2165.592236] Key type ._llcrypt unregistered [ 2169.801490] Key type ._llcrypt registered [ 2169.805062] Key type .llcrypt registered [ 2169.907982] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2174.972851] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules [ 2176.050360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2176.230260] alg: No test for adler32 (adler32-zlib) [ 2177.395107] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 2177.712528] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 2179.464994] Key type lgssc registered [ 2181.186625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2182.641922] LustreError: 90093:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2200.966233] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2203.701607] LustreError: 90093:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2209.241197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2209.258782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2210.740944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2214.782441] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2221.027514] LustreError: 91069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2221.042754] LustreError: 91069:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2221.988282] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 04:09:38 (1788595778) [ 2226.005219] LustreError: 91100:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2226.023978] LustreError: 91100:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2226.032485] LustreError: 91100:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 2226.379666] Lustre: server umount lustre-MDT0000 complete [ 2239.118671] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 2242.099364] Key type lgssc unregistered [ 2242.418819] LNet: 92594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2242.423644] LNetError: 92594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2242.451804] LNet: Removed LNI 192.168.203.114@tcp [ 2243.337403] Key type .llcrypt unregistered [ 2243.342446] Key type ._llcrypt unregistered [ 2259.114815] Key type ._llcrypt registered [ 2259.116642] Key type .llcrypt registered [ 2259.203726] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2259.858712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2259.890665] alg: No test for adler32 (adler32-zlib) [ 2261.062969] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 2261.399658] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 2263.080780] Key type lgssc registered [ 2264.497822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2273.688384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2273.711670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2275.301934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2279.817637] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2284.515052] LustreError: 93808:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2284.547244] LustreError: 93808:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2287.301371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2287.967155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2291.546610] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2298.256442] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2303.499184] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2309.657915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2310.545187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2316.856300] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2323.692161] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2365.649980] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 04:12:02 (1788595922) [ 2366.941566] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2368.469740] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 04:12:04 (1788595924) [ 2374.624779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2374.629351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2374.637252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2376.160075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2376.163190] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2376.183084] Lustre: Skipped 1 previous similar message [ 2377.792344] Lustre: server umount lustre-MDT0000 complete [ 2381.046933] LustreError: 97050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595938 with bad export cookie 16628606835037457380 [ 2381.062607] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2381.071542] LustreError: 97050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2381.280242] LustreError: 95657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.280911] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2381.449536] Lustre: server umount lustre-MDT0001 complete [ 2391.236521] Lustre: server umount lustre-OST0000 complete [ 2402.148126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.496971] LustreError: 98080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2402.514919] LustreError: 98080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2402.670700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2406.995734] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2407.911302] LustreError: 98081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2413.032833] LustreError: 98080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2414.670113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2415.327833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2419.710932] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2427.203645] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2433.567951] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2446.813731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2446.975497] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2446.984834] LustreError: 98069:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2447.013885] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2447.020594] LustreError: 100162:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2447.026589] LustreError: 100162:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2447.034853] LustreError: 100162:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2447.110881] Lustre: server umount lustre-OST0000 complete [ 2447.114534] LustreError: 100162:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2455.235765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2455.395071] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2455.408558] LustreError: 98069:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2455.433433] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2455.440908] LustreError: 100628:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2455.446836] LustreError: 100628:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2455.451047] LustreError: 100628:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2455.510601] Lustre: server umount lustre-OST0000 complete [ 2455.515151] LustreError: 100628:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2463.744078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2463.954901] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2463.965944] Lustre: Found index 0 for lustre-OST0000, updating log [ 2463.972922] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2464.068762] Lustre: lustre-OST0000: new disk, initializing [ 2464.073484] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2464.087293] Lustre: 101164:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2464.357619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2469.975983] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2473.453781] Lustre: Failing over lustre-OST0000 [ 2473.629262] Lustre: server umount lustre-OST0000 complete [ 2478.050375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2478.078516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2478.080869] Lustre: Skipped 2 previous similar messages [ 2483.168995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2483.179134] Lustre: Skipped 1 previous similar message [ 2487.327137] Lustre: server umount lustre-MDT0000 complete [ 2490.337315] LustreError: 98811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2490.347317] LustreError: 98811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2490.741679] LustreError: 98060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596048 with bad export cookie 16628606835037460124 [ 2490.742297] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2490.749305] LustreError: 98060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2491.041173] Lustre: server umount lustre-MDT0001 complete [ 2498.153356] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 04:14:14 (1788596054) [ 2511.681877] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2520.539737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2520.894983] LustreError: 103515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2521.029626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2525.083962] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2532.853887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2537.363600] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2539.775931] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2544.994295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2545.506463] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2546.537882] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2546.597188] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2546.598240] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2546.633949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2552.067795] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2560.716663] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2560.819425] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2560.907506] Lustre: lustre-OST0001: new disk, initializing [ 2560.912964] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2560.918631] Lustre: 105833:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2561.327454] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2561.339590] Lustre: Skipped 2 previous similar messages [ 2567.374649] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2569.859372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2569.870854] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2569.937664] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2575.128908] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2581.728296] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2585.793775] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.14@tcp (stopping) [ 2585.799978] Lustre: Skipped 2 previous similar messages [ 2590.176877] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2590.187889] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2590.211078] Lustre: Skipped 1 previous similar message [ 2592.066592] Lustre: server umount lustre-MDT0001 complete [ 2595.823213] LustreError: 103516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.846542] LustreError: 103516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2600.892282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2600.979951] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2600.984388] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2601.037524] Lustre: 103504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2602.733544] LustreError: 107414:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2602.741201] LustreError: 107414:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2602.752189] LustreError: 103508:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2602.824870] LustreError: 107419:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.114@tcp: cfg command failed: rc = -17 [ 2602.830242] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2602.830242] [ 2605.636655] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2607.091481] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2613.288271] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 04:16:09 (1788596169) [ 2630.076901] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2646.553191] Lustre: 109449:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2667.289446] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2670.372673] Lustre: 110584:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2749.921394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2749.930182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2749.955311] Lustre: Skipped 2 previous similar messages [ 2749.971295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2749.986228] Lustre: Skipped 4 previous similar messages [ 2754.995200] Lustre: server umount lustre-MDT0000 complete [ 2755.581248] LustreError: 103510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2755.619637] LustreError: 103510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 2759.095794] LustreError: 103496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596316 with bad export cookie 16628606835037461125 [ 2759.097212] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2759.103611] LustreError: 103496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2759.623750] Lustre: server umount lustre-MDT0001 complete [ 2774.024563] Lustre: server umount lustre-OST0000 complete [ 2777.058939] Lustre: 93119:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788596318/real 1788596318] req@ffff990d4ea41180 x1875478649850240/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788596334 ref 1 fl Rpc:EXNQr/200/ffffffff rc -5/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2786.444665] Lustre: server umount lustre-OST0001 complete [ 2791.257724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2800.083631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2807.799712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2816.613335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2835.645803] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2846.314136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2846.352096] Lustre: lustre-MDT0000: reset Object Index mappings [ 2846.896233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2846.902903] Lustre: Skipped 1 previous similar message [ 2851.585235] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2859.443377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2859.464279] Lustre: lustre-MDT0001: reset Object Index mappings [ 2864.177736] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2866.688696] Lustre: 115684:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2873.082612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2879.744916] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2885.234455] LustreError: 116038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2885.254344] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 2885.269175] LustreError: 116038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2887.080648] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2891.833858] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2891.842344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:386 to 0x280000401:418) [ 2891.843474] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2893.529796] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2900.263238] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2903.418525] Lustre: 117562:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2944.314209] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 04:21:40 (1788596500) [ 2972.661504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2972.681750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2972.698498] Lustre: Skipped 3 previous similar messages [ 2972.705412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2972.715948] Lustre: Skipped 4 previous similar messages [ 2977.624747] Lustre: server umount lustre-MDT0000 complete [ 2980.928354] LustreError: 116873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596538 with bad export cookie 16628606835037508200 [ 2980.934789] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2980.937372] LustreError: 116873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2987.473835] Lustre: server umount lustre-MDT0001 complete [ 2997.453091] Lustre: server umount lustre-OST0000 complete [ 3007.466592] Lustre: server umount lustre-OST0001 complete [ 3013.974975] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 3024.195906] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 3055.196236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3055.386979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3055.414506] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3055.510360] Lustre: lustre-MDT0000: new disk, initializing [ 3055.590740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3055.593856] Lustre: Skipped 3 previous similar messages [ 3055.616212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3059.808522] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3065.900963] Lustre: server umount lustre-MDT0000 complete [ 3074.180322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3074.433153] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3074.445620] Lustre: 121835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3074.464328] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3079.379470] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3084.669087] Lustre: server umount lustre-MDT0000 complete [ 3092.005856] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 04:24:08 (1788596648) [ 3093.250718] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3094.643578] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 04:24:11 (1788596651) [ 3109.206958] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 3118.293459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.775626] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3130.640677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3130.758096] Lustre: 123885:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3130.812296] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3130.817738] Lustre: Skipped 1 previous similar message [ 3130.887368] Lustre: lustre-MDT0001: new disk, initializing [ 3131.024161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3131.037943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3135.123289] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3139.330094] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3144.878410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3145.143756] Lustre: lustre-OST0000: new disk, initializing [ 3145.146746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3145.153485] Lustre: 125463:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3146.458142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3146.477976] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3146.599904] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3151.095156] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3159.516940] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3159.645148] Lustre: lustre-OST0001: new disk, initializing [ 3159.647946] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3159.652336] Lustre: 126332:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3161.582389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3161.591440] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3161.685622] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3164.817921] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3174.475344] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3177.823138] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3520.206072] Lustre: 132738:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3520.211495] Lustre: 132738:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3520.217597] Lustre: 132738:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3520.228246] Lustre: 132738:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3520.233640] Lustre: 132738:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3520.240457] Lustre: 132738:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3796.482366] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 04:35:52 (1788597352) [ 3811.816592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3811.821209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3811.831557] Lustre: Skipped 7 previous similar messages [ 3811.847712] Lustre: Skipped 11 previous similar messages [ 3816.935760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3816.944434] Lustre: Skipped 2 previous similar messages [ 3817.889427] Lustre: server umount lustre-MDT0000 complete [ 3821.197795] LustreError: 125462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597378 with bad export cookie 16628606835037573643 [ 3821.199020] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3821.232690] LustreError: 125462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3822.053955] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3822.068531] Lustre: Skipped 1 previous similar message [ 3822.071155] LustreError: 123891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3822.095394] LustreError: 123891:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 3826.082109] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3826.089400] Lustre: Skipped 3 previous similar messages [ 3827.717822] Lustre: server umount lustre-MDT0001 complete [ 3837.474930] Lustre: server umount lustre-OST0000 complete [ 3855.327120] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3855.563634] Lustre: server umount lustre-OST0001 complete [ 3863.526336] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 3873.365241] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 3901.343916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3901.505322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3901.523306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3901.582629] Lustre: lustre-MDT0000: new disk, initializing [ 3901.648927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3901.654475] Lustre: Skipped 5 previous similar messages [ 3901.669380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3905.629651] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3914.439925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3914.549515] Lustre: 135789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3914.592707] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3914.598472] Lustre: Skipped 1 previous similar message [ 3914.656821] Lustre: lustre-MDT0001: new disk, initializing [ 3914.769464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3914.782807] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3919.048533] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3927.004400] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3931.501206] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3937.572928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3937.852820] Lustre: lustre-OST0000: new disk, initializing [ 3937.856521] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3937.865560] Lustre: 137755:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3939.706359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3939.726512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3939.883283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3945.083066] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3954.067431] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3963.486278] Lustre: lustre-MDD0000: changelog on [ 3966.513961] Lustre: lustre-MDD0001: changelog on [ 5731.320314] Lustre: lustre-MDD0001: changelog off [ 5733.182925] Lustre: 135795:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5734.070181] Lustre: lustre-MDD0000: changelog off [ 5742.050874] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5742.054306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5742.061323] LustreError: Skipped 1 previous similar message [ 5742.064394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5742.083651] Lustre: Skipped 2 previous similar messages [ 5742.109840] Lustre: Skipped 1 previous similar message [ 5745.841039] Lustre: server umount lustre-OST0000 complete [ 5752.297682] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5752.312408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5752.317445] Lustre: Skipped 1 previous similar message [ 5755.382801] Lustre: server umount lustre-MDT0000 complete [ 5757.408450] LustreError: 136676:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5757.422795] LustreError: 136676:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5759.106372] LustreError: 135782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599316 with bad export cookie 16628606835038385594 [ 5759.109885] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5759.120671] LustreError: 135782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5759.440766] Lustre: server umount lustre-MDT0001 complete [ 5770.989515] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 5774.163302] Key type lgssc unregistered [ 5774.460726] LNet: 146878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5774.478096] LNetError: 146878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5774.495266] LNet: Removed LNI 192.168.203.114@tcp [ 5775.408852] Key type .llcrypt unregistered [ 5775.413241] Key type ._llcrypt unregistered [ 5796.927606] Key type ._llcrypt registered [ 5796.931690] Key type .llcrypt registered [ 5797.058829] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 5815.216511] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 5816.704996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5816.731122] alg: No test for adler32 (adler32-zlib) [ 5817.804225] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 5818.068404] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 5819.799421] Key type lgssc registered [ 5821.202597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5859.840704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5859.882300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5861.243339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5861.271449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5861.399904] Lustre: lustre-MDT0000: new disk, initializing [ 5861.533117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5861.556900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5865.237930] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5876.980928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5877.179919] Lustre: 150076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5877.303357] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5877.310892] Lustre: Skipped 1 previous similar message [ 5877.374532] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5877.488083] Lustre: lustre-MDT0001: new disk, initializing [ 5877.641858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5877.684137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5877.701615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5882.585766] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5893.409869] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5899.915924] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5907.391065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5907.748744] Lustre: lustre-OST0000: new disk, initializing [ 5907.753362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5907.763878] Lustre: 152044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5907.908606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5914.224530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5914.235540] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5914.365556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5915.608651] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5925.204334] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5931.897502] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_module obdecho/obdecho [ 5933.740530] LustreError: 153049:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5933.746774] LustreError: 153049:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5935.249637] LustreError: 153145:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5949.408280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5949.424071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5949.445629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5950.432426] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5950.458219] Lustre: Skipped 1 previous similar message [ 5950.461904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5950.466211] Lustre: Skipped 2 previous similar messages [ 5953.599724] Lustre: server umount lustre-MDT0000 complete [ 5957.414994] LustreError: 150066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599515 with bad export cookie 16021017394557372903 [ 5957.421878] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5957.432936] LustreError: 150066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5957.854081] Lustre: server umount lustre-MDT0001 complete [ 5972.952888] Lustre: server umount lustre-OST0000 complete [ 5996.397411] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 6008.124846] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 6043.205071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6043.510648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6043.560623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6043.677890] Lustre: lustre-MDT0000: new disk, initializing [ 6043.804709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6043.830052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6047.932401] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6057.849412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6057.995126] Lustre: 156863:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6058.064071] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6058.073934] Lustre: Skipped 1 previous similar message [ 6058.189843] Lustre: lustre-MDT0001: new disk, initializing [ 6058.301158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6058.347560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6058.362736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6063.141753] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6073.849417] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6080.533545] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6087.881386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6088.284528] Lustre: lustre-OST0000: new disk, initializing [ 6088.297090] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6088.321313] Lustre: 158829:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6088.444773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6095.412422] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6096.510230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6096.528536] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6096.668628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6106.624652] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6112.623174] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6112.858996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6118.045411] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6118.380734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6122.467709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6122.476702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6122.519380] Lustre: Skipped 1 previous similar message [ 6122.540935] Lustre: Skipped 1 previous similar message [ 6126.900614] Lustre: server umount lustre-OST0000 complete [ 6132.708505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6132.733980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6132.752190] Lustre: Skipped 1 previous similar message [ 6137.766194] Lustre: server umount lustre-MDT0000 complete [ 6137.837306] LustreError: 160194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6137.872206] LustreError: 160194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6141.772291] LustreError: 156853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599699 with bad export cookie 16021017394557373862 [ 6141.775085] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6141.783274] LustreError: 156853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6142.072330] Lustre: server umount lustre-MDT0001 complete [ 6145.886582] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 05:15:01 (1788599701) [ 6147.550941] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6149.520644] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 05:15:05 (1788599705) [ 6157.204741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6157.992295] LustreError: 161245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6158.121188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6162.962475] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6163.428850] LustreError: 161246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6163.469179] LustreError: 161246:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6167.527790] LustreError: 161245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6171.798284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6176.850815] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6184.600405] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6190.180542] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6196.449103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6196.646119] LustreError: 163064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6196.798994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6196.810234] Lustre: Skipped 1 previous similar message [ 6203.273079] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6210.561790] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6216.405077] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.14@tcp (stopping) [ 6218.216738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6218.221922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6218.238307] Lustre: Skipped 2 previous similar messages [ 6222.311503] Lustre: server umount lustre-MDT0000 complete [ 6223.329568] LustreError: 161241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6223.342023] LustreError: 161241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6225.909816] LustreError: 161225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599783 with bad export cookie 16021017394557374744 [ 6225.915413] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6225.926864] LustreError: 161225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6228.450573] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6228.467705] Lustre: Skipped 1 previous similar message [ 6232.343397] Lustre: server umount lustre-MDT0001 complete [ 6237.804065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6248.808156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6249.000349] LustreError: 165086:0:(llog.c:1655:llog_backup()) MGC192.168.203.114@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6249.010068] Lustre: 165086:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.114@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6256.098044] LustreError: 165091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6256.131270] LustreError: 165091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6290.018641] LustreError: 163075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6290.045226] LustreError: 163075:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 29 previous similar messages [ 6310.367325] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6310.373365] Lustre: 165086:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.114@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6350.916855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6352.938107] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6355.974633] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6357.998350] LustreError: 165091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6358.025071] LustreError: 165091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 47 previous similar messages [ 6365.080109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6372.332261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6377.681227] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6377.965333] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6385.008529] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6386.602651] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6392.681774] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6394.389643] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6413.792211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6413.797285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6413.809047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6413.809054] Lustre: Skipped 8 previous similar messages [ 6413.854236] Lustre: Skipped 1 previous similar message [ 6419.250534] Lustre: server umount lustre-MDT0000 complete [ 6422.959739] LustreError: 161225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599980 with bad export cookie 16021017394557381198 [ 6422.964508] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6422.977945] LustreError: 161225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6423.231821] Lustre: server umount lustre-MDT0001 complete [ 6437.331065] Lustre: server umount lustre-OST0000 complete [ 6460.503320] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 6471.208913] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 6504.304876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6504.587306] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6504.636875] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6504.745484] Lustre: lustre-MDT0000: new disk, initializing [ 6504.826131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6504.833281] Lustre: Skipped 1 previous similar message [ 6504.852902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6508.928924] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6519.326319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6519.481653] Lustre: 170192:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6519.789407] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6524.046806] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6533.015848] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6538.059156] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6543.502525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6543.799661] Lustre: lustre-OST0000: new disk, initializing [ 6543.806732] Lustre: Skipped 1 previous similar message [ 6543.810675] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6543.817694] Lustre: Skipped 2 previous similar messages [ 6543.823913] Lustre: 172162:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6549.411217] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6553.665766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6553.682889] Lustre: Skipped 1 previous similar message [ 6553.693996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6553.801037] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6558.512709] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6563.475858] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6563.762840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6568.299356] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6568.548830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6574.566242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6574.568772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6574.568877] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6574.597419] Lustre: Skipped 2 previous similar messages [ 6574.624180] Lustre: Skipped 3 previous similar messages [ 6576.649282] Lustre: server umount lustre-OST0000 complete [ 6586.812461] Lustre: server umount lustre-MDT0000 complete [ 6589.927637] LustreError: 170201:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6589.957711] LustreError: 170201:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [ 6590.499514] LustreError: 170185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600148 with bad export cookie 16021017394557398026 [ 6590.509819] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6590.514539] LustreError: 170185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6590.873575] Lustre: server umount lustre-MDT0001 complete [ 6600.008226] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 05:22:36 (1788600156) [ 6606.756067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6611.328299] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6619.368656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6623.912685] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6630.847410] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6636.037348] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6641.752240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6647.461913] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6654.704253] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6664.030850] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 05:23:40 (1788600220) [ 6670.817673] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6670.825326] LustreError: Skipped 2 previous similar messages [ 6670.838444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6670.852355] Lustre: Skipped 2 previous similar messages [ 6670.856296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6670.863227] Lustre: Skipped 3 previous similar messages [ 6674.441524] Lustre: server umount lustre-OST0000 complete [ 6684.492768] Lustre: server umount lustre-MDT0000 complete [ 6688.693232] LustreError: 174555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600246 with bad export cookie 16021017394557398908 [ 6688.700509] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6688.715208] LustreError: 174555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6688.991289] Lustre: server umount lustre-MDT0001 complete [ 6700.796032] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 6703.939450] Key type lgssc unregistered [ 6704.287212] LNet: 178298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6704.310330] LNetError: 178298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6704.334333] LNet: Removed LNI 192.168.203.114@tcp [ 6705.354503] Key type .llcrypt unregistered [ 6705.356454] Key type ._llcrypt unregistered [ 6725.436904] Key type ._llcrypt registered [ 6725.441620] Key type .llcrypt registered [ 6725.538841] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 6726.415463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6726.572319] alg: No test for adler32 (adler32-zlib) [ 6727.704352] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 6727.997584] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 6729.735181] Key type lgssc registered [ 6730.992629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6738.466640] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6738.508476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6754.015410] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6759.135554] LustreError: 179540:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.114@tcp: failed processing log, type 4: rc = -110 [ 6764.256850] LustreError: 179540:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6764.286729] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6764.295534] LustreError: MGC192.168.203.114@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6764.309300] LustreError: 179540:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6764.317727] LustreError: 179540:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 6764.324841] LustreError: 179540:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 6764.394726] Lustre: server umount lustre-OST0000 complete [ 6764.397987] LustreError: 179540:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6771.274482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6772.708061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6776.812481] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6782.958545] LustreError: 180039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6782.968116] LustreError: 180039:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6783.811736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6784.095885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6787.798298] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6794.550847] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6801.153714] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6807.423720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6807.651691] LustreError: 181856:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6807.660864] Lustre: 181856:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.114@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6807.848503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6813.455564] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6819.161622] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6823.396616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6823.402532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6823.428204] Lustre: Skipped 2 previous similar messages [ 6823.441747] Lustre: Skipped 1 previous similar message [ 6828.513993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6828.524195] Lustre: Skipped 3 previous similar messages [ 6833.634143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6833.644014] Lustre: Skipped 1 previous similar message [ 6835.167675] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6835.492546] Lustre: server umount lustre-MDT0000 complete [ 6838.759639] LustreError: 180036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6838.777245] LustreError: 180036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6838.805682] LustreError: 180773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600396 with bad export cookie 8797646548095381125 [ 6838.813876] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6838.820137] LustreError: 180773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6839.159820] Lustre: server umount lustre-MDT0001 complete [ 6853.263882] Lustre: server umount lustre-OST0000 complete [ 6861.991165] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 6870.213098] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 6906.278909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6906.485582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6906.535295] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6906.662806] Lustre: lustre-MDT0000: new disk, initializing [ 6906.755708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6906.767491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6910.881413] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6919.449130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6919.559235] Lustre: 185379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6919.626896] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6919.631470] Lustre: Skipped 1 previous similar message [ 6919.663760] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6919.720748] Lustre: lustre-MDT0001: new disk, initializing [ 6919.795938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6919.838107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6919.845254] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6923.613810] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6932.237688] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6937.463280] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6944.255287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6944.536623] Lustre: lustre-OST0000: new disk, initializing [ 6944.540738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6944.552595] Lustre: 187347:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6944.645353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6950.488820] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6953.023989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6953.030223] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6953.114250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6959.927614] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6965.239792] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6965.486190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6970.024622] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6970.262936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6973.930001] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6973.949763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6973.952463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6973.970856] Lustre: Skipped 1 previous similar message [ 6984.169633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6984.178323] Lustre: Skipped 3 previous similar messages [ 6986.209162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6986.391199] Lustre: server umount lustre-OST0000 complete [ 6991.329096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6991.332168] LustreError: Skipped 1 previous similar message [ 6991.338424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6991.351865] Lustre: Skipped 1 previous similar message [ 6994.405441] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6995.677143] Lustre: server umount lustre-MDT0000 complete [ 6998.722751] LustreError: 185372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600556 with bad export cookie 8797646548095381846 [ 6998.730326] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6998.735957] LustreError: 185372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6999.058684] Lustre: server umount lustre-MDT0001 complete [ 7006.349419] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 05:29:22 (1788600562) [ 7020.408875] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7022.598153] Key type lgssc unregistered [ 7022.890606] LNet: 190291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7022.897994] LNetError: 190291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7022.917167] LNet: Removed LNI 192.168.203.114@tcp [ 7023.657882] Key type .llcrypt unregistered [ 7023.660169] Key type ._llcrypt unregistered [ 7040.201677] Key type ._llcrypt registered [ 7040.204353] Key type .llcrypt registered [ 7040.302986] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7041.287679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7041.408340] alg: No test for adler32 (adler32-zlib) [ 7042.524272] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 7042.746874] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 7044.487256] Key type lgssc registered [ 7045.511682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7051.916633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7051.953513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7053.402312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7056.978120] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7063.532646] LustreError: 191507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7063.549817] LustreError: 191507:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7063.792081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7064.131769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7068.100239] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7075.669887] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7081.141261] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7089.213266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7089.492661] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7089.497853] LustreError: 193372:0:(llog.c:1696:llog_backup()) MGC192.168.203.114@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7089.520485] Lustre: 193372:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.114@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7089.667968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7096.099146] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7103.407252] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7108.585293] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7108.589667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7108.607542] Lustre: Skipped 2 previous similar messages [ 7108.632398] Lustre: Skipped 2 previous similar messages [ 7111.922366] Lustre: server umount lustre-MDT0000 complete [ 7113.701193] LustreError: 191504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7113.733848] LustreError: 191504:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7115.329944] LustreError: 191489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600673 with bad export cookie 11949374892483808237 [ 7115.351301] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7115.356904] LustreError: 191489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7116.835880] LustreError: 192237:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d9807330 x1875483662773120/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7117.021388] Lustre: server umount lustre-MDT0001 complete [ 7131.208273] Lustre: server umount lustre-OST0000 complete [ 7139.706966] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 7148.196359] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7178.493039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7178.790846] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7178.845643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7178.948242] Lustre: lustre-MDT0000: new disk, initializing [ 7179.029428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7179.044828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7182.402334] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7190.778497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7190.876131] Lustre: 196898:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7190.916311] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7190.918775] Lustre: Skipped 1 previous similar message [ 7190.922753] LustreError: 197681:0:(llog.c:1696:llog_backup()) MGC192.168.203.114@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7190.927568] LustreError: 197681:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7190.931367] Lustre: 197681:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.114@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7190.944104] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7190.948066] Lustre: Skipped 1 previous similar message [ 7191.004991] Lustre: lustre-MDT0001: new disk, initializing [ 7191.091675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7191.124420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7191.127297] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7194.231571] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7201.775180] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7207.755480] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7213.613700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7213.818143] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7213.820342] Lustre: Skipped 1 previous similar message [ 7213.823579] LustreError: 198840:0:(llog.c:1696:llog_backup()) MGC192.168.203.114@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7213.831414] LustreError: 198840:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7213.835753] Lustre: 198840:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.114@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7213.843750] Lustre: 198840:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7213.863710] Lustre: lustre-OST0000: new disk, initializing [ 7213.868846] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7213.874651] Lustre: 198863:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7213.989243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7220.719816] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7222.895403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7222.912673] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7223.010512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7229.928950] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7235.243737] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7235.475697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7240.499547] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7240.738278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7243.746129] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7243.747755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7243.764391] LustreError: Skipped 1 previous similar message [ 7243.765370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7243.828540] Lustre: Skipped 1 previous similar message [ 7248.875670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7248.879435] Lustre: Skipped 1 previous similar message [ 7253.989198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7253.997512] Lustre: Skipped 1 previous similar message [ 7257.055288] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7257.183250] Lustre: server umount lustre-OST0000 complete [ 7262.689137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7262.696710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7262.718980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7262.723882] Lustre: Skipped 1 previous similar message [ 7266.569509] Lustre: server umount lustre-MDT0000 complete [ 7269.348142] LustreError: 196909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7269.366341] LustreError: 196909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7270.263529] LustreError: 196890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600828 with bad export cookie 11949374892483808958 [ 7270.265924] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7270.274780] LustreError: 196890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7270.608730] Lustre: server umount lustre-MDT0001 complete [ 7278.256231] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 05:33:54 (1788600834) [ 7292.657659] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7301.582668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7301.923207] LustreError: 201989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7301.976217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7306.113877] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7307.232714] LustreError: 201990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7313.416558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7318.172884] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7320.543901] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7326.338039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7331.915403] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7339.723644] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7339.916813] Lustre: lustre-OST0001: new disk, initializing [ 7339.920239] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7339.933148] Lustre: 204295:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7340.075687] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7340.079299] Lustre: Skipped 2 previous similar messages [ 7345.704206] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7347.314403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7347.323625] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7347.397408] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7355.233718] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7361.785757] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7363.456386] Lustre: DEBUG MARKER: ADD OST3 [ 7367.996700] loop: module loaded [ 7368.068477] loop0: detected capacity change from 0 to 204800000 [ 7368.330394] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7368.345176] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7368.370427] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7373.837640] loop0: detected capacity change from 0 to 204800000 [ 7375.048394] Lustre: DEBUG MARKER: STOP OST3 [ 7379.049188] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7379.231666] Lustre: lustre-OST0003: new disk, initializing [ 7379.239468] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7379.257256] Lustre: 206253:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7380.340709] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7380.439128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7380.450486] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7380.498055] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7382.371082] LustreError: 206402:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7382.788759] Lustre: Failing over lustre-OST0003 [ 7382.868453] Lustre: server umount lustre-OST0003 complete [ 7384.704732] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7385.569423] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7385.580216] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7385.595199] Lustre: Skipped 1 previous similar message [ 7385.618082] LustreError: 203474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7385.639466] LustreError: 203474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7393.008985] Lustre: DEBUG MARKER: START OST3 again [ 7398.468394] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7398.753616] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7399.787782] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7399.962841] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7399.963242] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7403.776920] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7422.709771] Lustre: Failing over lustre-OST0003 [ 7422.787352] Lustre: server umount lustre-OST0003 complete [ 7424.490741] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7424.493817] LustreError: 203474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7424.500811] Lustre: Skipped 2 previous similar messages [ 7424.530511] LustreError: 203474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 7439.858489] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7439.862417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7439.882297] Lustre: Skipped 3 previous similar messages [ 7439.893262] Lustre: Skipped 5 previous similar messages [ 7447.691279] Lustre: server umount lustre-MDT0000 complete [ 7450.654530] LustreError: 201971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788601008 with bad export cookie 11949374892483809840 [ 7450.657902] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7450.675143] LustreError: 201971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7457.053715] Lustre: server umount lustre-MDT0001 complete [ 7460.246283] Lustre: server umount lustre-OST0000 complete [ 7463.743378] Lustre: server umount lustre-OST0001 complete [ 7469.375162] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 7476.394683] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7511.720395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7511.946354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7511.994325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7512.055843] Lustre: lustre-MDT0000: new disk, initializing [ 7512.134267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7512.138743] Lustre: Skipped 2 previous similar messages [ 7512.149812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7516.763733] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7528.063359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7528.277431] Lustre: 211754:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7528.368163] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7528.374685] Lustre: Skipped 1 previous similar message [ 7528.481613] Lustre: lustre-MDT0001: new disk, initializing [ 7528.625609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7528.643930] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7534.737873] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7544.320892] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7548.692985] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7554.950834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7555.192399] Lustre: 213722:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7556.421392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7556.529873] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7560.413617] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7568.057656] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7571.810562] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7572.018546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7575.576644] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7575.763673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7582.178235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7582.183081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7582.184391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7582.184397] Lustre: Skipped 6 previous similar messages [ 7582.203974] Lustre: Skipped 3 previous similar messages [ 7583.789560] Lustre: server umount lustre-OST0000 complete [ 7593.221945] Lustre: server umount lustre-MDT0000 complete [ 7596.616049] LustreError: 213721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788601154 with bad export cookie 11949374892483812136 [ 7596.620978] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7596.631125] LustreError: 213721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7596.999351] Lustre: server umount lustre-MDT0001 complete [ 7600.304513] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 05:39:16 (1788601156) [ 7613.900340] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 7623.750456] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7658.186840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7658.497995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7658.535051] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7658.540598] Lustre: Skipped 1 previous similar message [ 7658.636786] Lustre: lustre-MDT0000: new disk, initializing [ 7658.641754] Lustre: Skipped 1 previous similar message [ 7658.731070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7658.735893] Lustre: Skipped 2 previous similar messages [ 7658.749337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7658.757164] Lustre: Skipped 1 previous similar message [ 7663.285288] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7673.752228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7673.889536] Lustre: 218326:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7674.131260] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7678.178701] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7687.651524] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7692.545549] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7698.935199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7699.152573] Lustre: 220296:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7700.996337] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7705.392423] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7714.040710] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7718.352477] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7718.557976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7723.238096] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7723.387577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7726.563718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7726.566169] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7726.567674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7726.567680] Lustre: Skipped 5 previous similar messages [ 7726.582038] Lustre: Skipped 3 previous similar messages [ 7726.599642] LustreError: Skipped 1 previous similar message [ 7739.359550] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7739.593473] Lustre: server umount lustre-OST0000 complete [ 7746.017323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7749.725390] Lustre: server umount lustre-MDT0000 complete [ 7752.172815] LustreError: 220091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7752.195360] LustreError: 220091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 7753.917832] LustreError: 218320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788601311 with bad export cookie 11949374892483813018 [ 7753.931226] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7753.942315] LustreError: 218320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7754.236782] Lustre: server umount lustre-MDT0001 complete [ 7762.105964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7766.963394] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7775.294388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7779.667845] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7786.473658] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7792.748955] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7801.493787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7809.104434] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7815.466124] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7825.903470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7825.906254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7825.918761] Lustre: Skipped 3 previous similar messages [ 7825.923136] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7825.928601] Lustre: Skipped 9 previous similar messages [ 7826.514304] Lustre: server umount lustre-OST0000 complete [ 7836.565909] Lustre: server umount lustre-MDT0000 complete [ 7840.208938] LustreError: 222497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788601397 with bad export cookie 11949374892483813900 [ 7840.221906] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7840.233089] LustreError: 222497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7840.603963] Lustre: server umount lustre-MDT0001 complete [ 7953.399613] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7957.051788] Key type lgssc unregistered [ 7957.464836] LNet: 226182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7957.481969] LNetError: 226182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7957.510909] LNet: Removed LNI 192.168.203.114@tcp [ 7958.492155] Key type .llcrypt unregistered [ 7958.498434] Key type ._llcrypt unregistered [ 7984.115429] Key type ._llcrypt registered [ 7984.117989] Key type .llcrypt registered [ 7984.198560] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 7998.247968] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7999.307626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7999.372250] alg: No test for adler32 (adler32-zlib) [ 8000.488576] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 8000.771847] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 8002.495210] Key type lgssc registered [ 8003.485783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8036.009817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8036.060553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8037.319344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8037.381907] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8037.554617] Lustre: lustre-MDT0000: new disk, initializing [ 8037.653968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8037.677099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8042.126899] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8051.965354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8052.097689] Lustre: 229531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8052.155172] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8052.158377] Lustre: Skipped 1 previous similar message [ 8052.229232] Lustre: lustre-MDT0001: new disk, initializing [ 8052.308163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8052.349331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8052.357569] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8056.684274] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8066.266888] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8072.234961] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8079.321950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8079.738870] Lustre: lustre-OST0000: new disk, initializing [ 8079.747334] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8079.759275] Lustre: 231503:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8079.856222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8086.334820] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8087.136562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8087.155118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8087.325187] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8096.299177] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8100.777047] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8101.009717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8105.563458] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8105.823777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8108.004085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8108.007040] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8108.035779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8108.040096] LustreError: Skipped 1 previous similar message [ 8108.063278] Lustre: Skipped 1 previous similar message [ 8113.133537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8113.146971] Lustre: Skipped 1 previous similar message [ 8113.818518] Lustre: server umount lustre-OST0000 complete [ 8118.252973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8118.264045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8118.282676] Lustre: Skipped 2 previous similar messages [ 8118.307873] Lustre: Skipped 1 previous similar message [ 8123.361367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8123.368261] Lustre: Skipped 1 previous similar message [ 8124.070977] Lustre: server umount lustre-MDT0000 complete [ 8128.043671] LustreError: 229524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788601685 with bad export cookie 15914708924516132508 [ 8128.050346] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8128.060378] LustreError: 229524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8128.316305] Lustre: server umount lustre-MDT0001 complete [ 8149.338401] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8160.959434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8161.544745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8166.881184] LustreError: 234436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8166.891085] LustreError: 234436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8167.131269] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8172.008708] LustreError: 234437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8175.875862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8176.287035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8180.939588] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8183.849850] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8190.420869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8190.823845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8197.019280] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8205.825770] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8206.094412] Lustre: lustre-OST0001: new disk, initializing [ 8206.101092] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8206.110946] Lustre: 236746:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8211.585150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8211.599478] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8211.731693] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8213.227161] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8224.227400] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8236.275305] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8248.460272] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 05:50:04 (1788601804) [ 8252.897272] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8252.904683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8252.909223] Lustre: Skipped 3 previous similar messages [ 8252.909530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8252.914671] Lustre: Skipped 3 previous similar messages [ 8256.685109] Lustre: server umount lustre-MDT0000 complete [ 8258.025557] LustreError: 235451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8258.057475] LustreError: 235451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8259.952434] LustreError: 237514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788601817 with bad export cookie 15914708924516133397 [ 8259.953448] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8259.962542] LustreError: 237514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8260.283041] Lustre: server umount lustre-MDT0001 complete [ 8274.304934] Lustre: server umount lustre-OST0000 complete [ 8286.745090] Lustre: server umount lustre-OST0001 complete [ 8293.054420] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 8303.154296] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8339.253120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8339.690970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8339.769306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8339.890401] Lustre: lustre-MDT0000: new disk, initializing [ 8340.015589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8340.018525] Lustre: Skipped 1 previous similar message [ 8340.034059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8345.823593] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8358.190758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8358.396229] Lustre: 240691:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8358.504616] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8358.511553] Lustre: Skipped 1 previous similar message [ 8358.693827] Lustre: lustre-MDT0001: new disk, initializing [ 8358.958970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8358.985795] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8364.982488] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8375.309183] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8380.251349] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8386.223161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8386.463446] Lustre: lustre-OST0000: new disk, initializing [ 8386.466878] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8386.471771] Lustre: 242661:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8386.567993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8386.586841] Lustre: Skipped 1 previous similar message [ 8387.942025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8387.956630] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8388.118758] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8392.997683] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8400.973174] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8405.399469] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8405.635891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8410.620540] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8410.808933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8413.664675] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8413.675314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8413.693699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8419.053593] Lustre: server umount lustre-OST0000 complete [ 8423.904696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8423.918605] Lustre: Skipped 2 previous similar messages [ 8429.756866] Lustre: server umount lustre-MDT0000 complete [ 8433.991885] LustreError: 240685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788601991 with bad export cookie 15914708924516135532 [ 8434.009293] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8434.012777] LustreError: 240685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8434.390766] Lustre: server umount lustre-MDT0001 complete [ 8443.534159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8444.158635] LustreError: 244877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8449.225877] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8449.505436] LustreError: 244878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8458.728118] LustreError: 244877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8458.759160] LustreError: 244877:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8460.302714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8460.916355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8460.926618] Lustre: Skipped 1 previous similar message [ 8467.531245] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8476.366714] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8482.742966] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8490.167426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8498.787401] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8506.238494] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8652.773280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8652.774664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8652.790644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8652.796926] LustreError: Skipped 1 previous similar message [ 8652.809385] Lustre: Skipped 8 previous similar messages [ 8658.993551] Lustre: server umount lustre-OST0000 complete [ 8665.569373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8669.515746] Lustre: server umount lustre-MDT0000 complete [ 8673.248731] LustreError: 245795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8673.270846] LustreError: 245795:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8673.420108] LustreError: 244858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788602231 with bad export cookie 15914708924516136414 [ 8673.433409] LustreError: 244858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8673.449538] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8673.878938] Lustre: server umount lustre-MDT0001 complete [ 8776.181440] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 8779.130585] Key type lgssc unregistered [ 8779.441962] LNet: 248483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8779.449433] LNetError: 248483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8779.476752] LNet: Removed LNI 192.168.203.114@tcp [ 8780.282142] Key type .llcrypt unregistered [ 8780.284907] Key type ._llcrypt unregistered [ 8804.820057] Key type ._llcrypt registered [ 8804.822177] Key type .llcrypt registered [ 8804.909886] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 8821.003666] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8822.179158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8822.230924] alg: No test for adler32 (adler32-zlib) [ 8823.319132] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 8823.651238] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 8825.415369] Key type lgssc registered [ 8826.792500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8870.214458] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8878.750585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8878.819801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8879.993256] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8880.020266] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8880.096727] Lustre: lustre-MDT0000: new disk, initializing [ 8880.178715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8880.193083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8883.669623] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8893.207416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8893.368227] Lustre: 252553:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8893.420286] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8893.431271] Lustre: Skipped 1 previous similar message [ 8893.546685] Lustre: lustre-MDT0001: new disk, initializing [ 8893.665633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8893.714220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8893.732358] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8899.688472] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8904.800383] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8911.258568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8911.577399] Lustre: lustre-OST0000: new disk, initializing [ 8911.581089] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8911.586315] Lustre: 254194:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8911.657946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8918.187501] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8919.142281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8919.153910] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8919.272806] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8929.623845] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8929.882490] Lustre: lustre-OST0001: new disk, initializing [ 8929.886288] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8929.896435] Lustre: 255068:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8930.008526] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8935.284647] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8936.529841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8936.536419] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8936.673847] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8946.902250] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8954.776326] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8967.652626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8967.661121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8967.682992] Lustre: Skipped 2 previous similar messages [ 8967.692601] Lustre: Skipped 1 previous similar message [ 8972.108888] Lustre: server umount lustre-MDT0000 complete [ 8975.693489] LustreError: 252544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788602533 with bad export cookie 10975391195408270995 [ 8975.713119] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8975.716270] LustreError: 252544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8976.067556] Lustre: server umount lustre-MDT0001 complete [ 8989.869443] Lustre: server umount lustre-OST0000 complete [ 9003.891248] Lustre: server umount lustre-OST0001 complete [ 9016.155375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9025.195683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9032.651304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9040.325878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9057.213478] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 9066.896365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9066.916450] Lustre: lustre-MDT0000: reset Object Index mappings [ 9067.370340] LustreError: 260075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9067.390370] LustreError: 260075:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9067.455804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9071.185232] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9072.611246] LustreError: 260076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9077.735658] LustreError: 260075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9078.701567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9078.724727] Lustre: lustre-MDT0001: reset Object Index mappings [ 9079.135922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9083.306203] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9086.018516] Lustre: 261218:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9092.238698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9093.623551] LustreError: 261572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9093.636916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9098.525795] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9098.726724] LustreError: 261573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9106.087902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9106.321316] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9106.331316] Lustre: Skipped 1 previous similar message [ 9111.178628] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9111.559490] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9117.773731] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9121.190698] Lustre: 263088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9126.893684] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9126.924213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9126.925660] Lustre: Skipped 4 previous similar messages [ 9126.972776] Lustre: Skipped 3 previous similar messages [ 9128.118703] Lustre: server umount lustre-MDT0000 complete [ 9131.606842] LustreError: 261220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788602689 with bad export cookie 10975391195408277785 [ 9131.620226] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9131.626391] LustreError: 261220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9131.930556] Lustre: server umount lustre-MDT0001 complete [ 9145.731479] Lustre: server umount lustre-OST0000 complete [ 9147.362215] Lustre: 250057:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788602689/real 1788602689] req@ffff990e68d12d80 x1875485530382208/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788602705 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9147.392023] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9149.163672] Lustre: server umount lustre-OST0001 complete [ 9167.140535] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 06:05:23 (1788602723) [ 9183.110857] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 9191.317233] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 9219.914393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9220.094732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9220.138287] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9220.212455] Lustre: lustre-MDT0000: new disk, initializing [ 9220.279776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9220.295315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9224.238127] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9233.282720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9233.382405] Lustre: 267342:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9233.433192] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9233.438063] Lustre: Skipped 1 previous similar message [ 9233.523032] Lustre: lustre-MDT0001: new disk, initializing [ 9233.640580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9233.651781] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9237.776986] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9247.581829] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9252.453952] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9257.463763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9257.644978] Lustre: lustre-OST0000: new disk, initializing [ 9257.648157] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9257.653317] Lustre: 269306:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9259.257651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9259.273773] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9259.392058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9262.739826] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9271.626812] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9277.542280] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9277.919754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9283.030740] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9283.218646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9285.092261] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9285.101828] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9285.105552] LustreError: Skipped 1 previous similar message [ 9285.126143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9285.133062] Lustre: Skipped 1 previous similar message [ 9290.227970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9290.247153] Lustre: Skipped 1 previous similar message [ 9291.130802] Lustre: server umount lustre-OST0000 complete [ 9295.332576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9295.344240] Lustre: Skipped 1 previous similar message [ 9295.352479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9295.363747] Lustre: Skipped 1 previous similar message [ 9300.354360] Lustre: server umount lustre-MDT0000 complete [ 9300.450234] LustreError: 269771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9300.469636] LustreError: 269771:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9303.513159] LustreError: 267331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788602861 with bad export cookie 10975391195408279878 [ 9303.516083] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9303.528902] LustreError: 267331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9303.810765] Lustre: server umount lustre-MDT0001 complete [ 9318.406681] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 9326.224895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9326.523652] LustreError: 272242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9326.562835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9326.568170] Lustre: Skipped 2 previous similar messages [ 9330.065293] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9336.558241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9340.020755] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9342.234674] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9348.522754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9354.928399] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9363.277494] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9363.530648] Lustre: lustre-OST0001: new disk, initializing [ 9363.533608] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9363.546691] Lustre: 274549:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9365.300451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9365.315628] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9365.533307] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9371.665840] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9381.587606] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9385.250561] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9389.996210] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9390.106469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9401.151115] Lustre: server umount lustre-MDT0000 complete [ 9401.322618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9401.325266] LustreError: 272237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9401.338079] Lustre: Skipped 4 previous similar messages [ 9401.370522] LustreError: 272237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9403.776034] LustreError: 272222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788602961 with bad export cookie 10975391195408280760 [ 9403.781351] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9403.783801] LustreError: 272222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9404.014216] Lustre: server umount lustre-MDT0001 complete [ 9417.049129] Lustre: server umount lustre-OST0000 complete [ 9430.923690] Lustre: server umount lustre-OST0001 complete [ 9443.322428] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 9451.821370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9456.255207] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9463.080126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9463.373829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9463.378123] Lustre: Skipped 4 previous similar messages [ 9466.222747] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9468.365201] Lustre: 279032:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9473.123063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9474.538695] LustreError: 279388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9474.559526] LustreError: 279388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9477.620988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9477.843816] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9483.642989] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9488.430884] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9489.394608] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9493.954974] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9496.621418] Lustre: 280908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9500.297705] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9500.323898] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9509.809232] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 06:11:06 (1788603066) [ 9524.704431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9524.717052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9524.726494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9526.214908] Lustre: server umount lustre-MDT0000 complete [ 9529.433323] LustreError: 277872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788603087 with bad export cookie 10975391195408287144 [ 9529.438961] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9529.442792] LustreError: 277872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9531.723247] Lustre: server umount lustre-MDT0001 complete [ 9549.279126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9549.337101] Lustre: server umount lustre-OST0000 complete [ 9559.263421] Lustre: server umount lustre-OST0001 complete [ 9564.969201] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 9573.734523] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 9613.951355] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 9622.322738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9622.469849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9622.491572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9622.580969] Lustre: lustre-MDT0000: new disk, initializing [ 9622.654834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9626.247569] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9634.861583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9634.983970] Lustre: 285432:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9635.240209] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9639.120895] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9649.202433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9649.504497] Lustre: 287067:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9651.554222] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9654.949658] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9663.599974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9663.773647] Lustre: 287939:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9665.166217] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9669.526867] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9677.932888] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9681.207664] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9681.213508] Lustre: Skipped 1 previous similar message [ 9692.581210] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 06:14:08 (1788603248) [ 9696.228314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9696.241287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9696.241826] Lustre: Skipped 6 previous similar messages [ 9696.246529] Lustre: Skipped 6 previous similar messages [ 9700.804716] Lustre: server umount lustre-MDT0000 complete [ 9701.346653] LustreError: 285444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9701.374100] LustreError: 285444:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 9703.756340] LustreError: 285425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788603261 with bad export cookie 10975391195408295089 [ 9703.763438] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9703.768965] LustreError: 285425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9704.053988] Lustre: server umount lustre-MDT0001 complete [ 9716.384175] Lustre: server umount lustre-OST0000 complete [ 9729.269755] Lustre: server umount lustre-OST0001 complete [ 9736.898370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9737.384625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9737.391174] Lustre: Skipped 6 previous similar messages [ 9741.032702] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9748.463387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9752.364822] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9758.737925] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9763.492740] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9769.035680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9775.246603] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9781.679480] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9799.749321] loop1: detected capacity change from 0 to 204800000 [ 9799.933760] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9799.955984] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9799.963670] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9804.538755] loop1: detected capacity change from 0 to 204800000 [ 9809.637218] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9809.724225] LustreError: 290173:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs23969: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9809.744489] LustreError: 290173:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs23969 [ 9809.747084] LustreError: 290173:0:(mgs_handler.c:517:mgs_target_reg()) fs23969-MDT0001: Error checking target index: rc = -13 [ 9809.752294] LustreError: fs23969-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9809.756392] LustreError: 293905:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [ 9809.759849] LustreError: 293905:0:(tgt_mount.c:2132:server_put_super()) no obd fs23969-MDT0001 [ 9809.763196] LustreError: 293905:0:(tgt_mount.c:116:server_deregister_mount()) fs23969-MDT0001 not registered [ 9809.785183] Lustre: server umount fs23969-MDT0001 complete [ 9809.788450] LustreError: 293905:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9816.921026] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9817.095251] Lustre: Setting parameter fs23969-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs23969-MDT0001 [ 9817.154812] Lustre: srv-fs23969-MDT0001: No data found on store. Initialize space. [ 9817.159724] Lustre: Skipped 4 previous similar messages [ 9817.237502] Lustre: fs23969-MDT0001: new disk, initializing [ 9817.241954] Lustre: Skipped 3 previous similar messages [ 9817.421203] Lustre: srv-fs23969-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9817.437232] Lustre: 294372:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs23969-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9817.454844] LustreError: 294372:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs23969-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9821.041714] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9827.143262] Lustre: server umount fs23969-MDT0001 complete [ 9839.074621] loop1: detected capacity change from 0 to 204800000 [ 9844.013802] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9844.469767] Lustre: 295939:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs23969-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9844.485727] LustreError: 295939:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs23969-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9848.497444] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9852.174713] Lustre: server umount fs23969-MDT0001 complete [ 9859.574955] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 9861.605386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9861.624127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9861.640983] Lustre: Skipped 2 previous similar messages [ 9861.658228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9861.661653] Lustre: Skipped 3 previous similar messages [ 9866.198398] Lustre: server umount lustre-MDT0000 complete [ 9873.269036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9873.385893] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9876.677367] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9879.023810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9882.398844] loop1: detected capacity change from 0 to 204800000 [ 9886.047928] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9886.310661] Lustre: 298382:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs23969-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9886.321103] LustreError: 298382:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs23969-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9889.188463] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9892.792478] Lustre: server umount fs23969-MDT0001 complete [ 9912.809171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9925.087147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9925.258295] Lustre: server umount lustre-OST0000 complete [ 9930.208515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9930.213494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9930.214979] LustreError: Skipped 1 previous similar message [ 9930.227559] Lustre: Skipped 12 previous similar messages [ 9934.766219] Lustre: server umount lustre-MDT0000 complete [ 9937.921395] LustreError: 290167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788603495 with bad export cookie 10975391195408297770 [ 9937.928675] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9937.932908] LustreError: 290167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9938.243796] Lustre: server umount lustre-MDT0001 complete [ 9947.928625] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 9950.447622] Key type lgssc unregistered [ 9950.759743] LNet: 300724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9950.769245] LNetError: 300724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9950.796741] LNet: Removed LNI 192.168.203.114@tcp [ 9951.544145] Key type .llcrypt unregistered [ 9951.546462] Key type ._llcrypt unregistered [ 9968.017917] Key type ._llcrypt registered [ 9968.019943] Key type .llcrypt registered [ 9968.087358] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 9968.834190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9968.846023] alg: No test for adler32 (adler32-zlib) [ 9969.855570] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 9970.038020] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 9971.679260] Key type lgssc registered [ 9972.574772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9978.084703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9978.113806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9979.432640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9982.293357] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9988.599434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9989.024107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9992.488929] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9998.299572] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10003.507717] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10009.094710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10009.345732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10014.439662] LustreError: 303822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10014.450802] LustreError: 303822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10014.957633] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10019.818622] LustreError: 304145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10020.932892] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10024.934678] LustreError: 303824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10024.961917] LustreError: 303824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10027.716756] LustreError: 303822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10032.817257] LustreError: 304145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10032.840097] LustreError: 304145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10039.807142] loop1: detected capacity change from 0 to 204800000 [10039.968931] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10039.984955] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10040.000524] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10043.054511] LustreError: 304505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10043.072738] LustreError: 304505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10044.034427] loop1: detected capacity change from 0 to 204800000 [10048.153332] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10048.276102] Lustre: fs18224-OST0000: new disk, initializing [10048.281173] Lustre: srv-fs18224-OST0000: No data found on store. Initialize space. [10048.288471] Lustre: 305727:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10048.367283] Lustre: fs18224-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10053.395620] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10059.897189] Lustre: server umount fs18224-OST0000 complete [10060.776200] LustreError: 304505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10060.792921] LustreError: 304505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [10069.713783] loop1: detected capacity change from 0 to 204800000 [10074.097214] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10074.292639] Lustre: fs18224-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10079.198742] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10086.317153] loop2: detected capacity change from 0 to 204800000 [10086.531030] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10086.553972] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10086.583715] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10090.540426] loop2: detected capacity change from 0 to 204800000 [10094.252970] LustreError: 303824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10094.270115] LustreError: 303824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [10095.750876] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10095.794203] LustreError: fs18224-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10095.812807] LustreError: fs18224-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10095.830310] LustreError: 308764:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [10095.837924] LustreError: 308764:0:(tgt_mount.c:2132:server_put_super()) no obd fs18224-OST0001 [10095.844167] LustreError: 308764:0:(tgt_mount.c:116:server_deregister_mount()) fs18224-OST0001 not registered [10095.874122] Lustre: server umount fs18224-OST0001 complete [10095.876587] LustreError: 308764:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10101.745643] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10106.698959] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10106.863275] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10106.939183] Lustre: fs18224-OST0001: new disk, initializing [10106.945691] Lustre: srv-fs18224-OST0001: No data found on store. Initialize space. [10106.950788] Lustre: 309317:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10107.061569] Lustre: fs18224-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10111.973349] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10119.458594] Lustre: server umount fs18224-OST0000 complete [10128.383565] Lustre: server umount fs18224-OST0001 complete [10140.569521] loop1: detected capacity change from 0 to 204800000 [10144.338729] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10144.502112] Lustre: fs18224-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10148.549456] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10153.882291] loop2: detected capacity change from 0 to 204800000 [10158.475872] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10160.819057] LustreError: 304505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10160.845883] LustreError: 304505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [10163.693380] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10171.113831] Lustre: server umount fs18224-OST0000 complete [10178.035892] Lustre: server umount fs18224-OST0001 complete [10187.749066] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10187.754673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10187.767213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10188.771434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10188.774653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10188.787424] Lustre: Skipped 1 previous similar message [10193.346602] Lustre: server umount lustre-OST0000 complete [10199.009331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10199.010544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10199.011500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10199.020468] Lustre: Skipped 1 previous similar message [10199.040851] LustreError: Skipped 1 previous similar message [10204.129556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10204.134492] Lustre: Skipped 2 previous similar messages [10209.255087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10209.264654] Lustre: Skipped 1 previous similar message [10210.271180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10210.445508] Lustre: server umount lustre-MDT0000 complete [10213.243282] LustreError: 301976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788603771 with bad export cookie 17576434785495969915 [10213.253638] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10213.259121] LustreError: 301976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10213.539166] Lustre: server umount lustre-MDT0001 complete [10221.705857] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [10223.671934] Key type lgssc unregistered [10223.949609] LNet: 315236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10223.964785] LNetError: 315236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10223.982324] LNet: Removed LNI 192.168.203.114@tcp [10224.631149] Key type .llcrypt unregistered [10224.634099] Key type ._llcrypt unregistered [10247.854636] Key type ._llcrypt registered [10247.857188] Key type .llcrypt registered [10247.947720] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [10261.624843] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [10262.386297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10262.473393] alg: No test for adler32 (adler32-zlib) [10263.555271] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [10263.753702] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [10265.393174] Key type lgssc registered [10266.231702] Lustre: Echo OBD driver; http://www.lustre.org/ [10298.262809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10298.289794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10299.530590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10299.559810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10299.623577] Lustre: lustre-MDT0000: new disk, initializing [10299.755878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10299.782753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10303.649598] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10313.709223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10313.794157] Lustre: 319092:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10313.862574] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10313.868524] Lustre: Skipped 1 previous similar message [10313.932219] Lustre: lustre-MDT0001: new disk, initializing [10313.999840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10314.039893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10314.048363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10318.075314] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10325.642987] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10330.706830] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10337.125870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10337.281971] Lustre: lustre-OST0000: new disk, initializing [10337.285310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10337.290437] Lustre: 321059:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10337.331160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10341.436340] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10345.010482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10345.030375] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10345.109351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10349.796302] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10352.058116] Lustre: Mounted lustre-client - version 2.17.58_2_gae7f7f4 [10352.893229] Lustre: Unmounted lustre-client [10362.338410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10362.342585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10362.359874] Lustre: Skipped 1 previous similar message [10362.374851] Lustre: Skipped 1 previous similar message [10367.512541] Lustre: server umount lustre-OST0000 complete [10372.579892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10372.586902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10372.619788] Lustre: Skipped 1 previous similar message [10372.629052] Lustre: Skipped 1 previous similar message [10377.697273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10377.704264] Lustre: Skipped 1 previous similar message [10382.822164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10382.840016] Lustre: Skipped 1 previous similar message [10386.399124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10386.721981] Lustre: server umount lustre-MDT0000 complete [10390.419489] LustreError: 319082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788603948 with bad export cookie 5027347718703490082 [10390.421675] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10390.430181] LustreError: 319082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10390.753303] Lustre: server umount lustre-MDT0001 complete [10401.257991] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [10404.323772] Key type lgssc unregistered [10406.112615] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [10406.687632] LNet: 323218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10406.701573] LNetError: 323218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10483.564714] LNet: Removed LNI 192.168.203.114@tcp [10484.384858] Key type .llcrypt unregistered [10484.387500] Key type ._llcrypt unregistered [10500.836660] Key type ._llcrypt registered [10500.841712] Key type .llcrypt registered [10500.983063] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [10515.848213] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [10516.885368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10516.987895] alg: No test for adler32 (adler32-zlib) [10518.024544] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [10518.251678] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [10519.943760] Key type lgssc registered [10521.074944] Lustre: Echo OBD driver; http://www.lustre.org/ [10549.161964] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 06:28:25 (1788604105) [10554.250442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10554.274840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10555.547338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10555.578973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10555.649240] Lustre: lustre-MDT0000: new disk, initializing [10555.743868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10555.760770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10560.382551] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10567.791896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10567.854990] Lustre: 326434:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10567.931979] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10567.936598] Lustre: Skipped 1 previous similar message [10567.938207] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10568.003429] Lustre: lustre-MDT0001: new disk, initializing [10568.079474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10568.101618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10568.108379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10571.111429] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10578.907834] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10583.590874] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10588.568244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10588.823237] Lustre: lustre-OST0000: new disk, initializing [10588.843308] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10588.852963] Lustre: 328403:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10588.906959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10594.560431] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10596.379680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10596.398410] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10596.478766] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10602.453052] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10616.801505] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10616.811630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10616.829246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10617.313465] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10617.322159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10620.735288] Lustre: server umount lustre-OST0000 complete [10624.480488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10624.488806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10624.501824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10624.506206] Lustre: Skipped 2 previous similar messages [10627.552544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10627.559840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10629.688544] Lustre: server umount lustre-MDT0000 complete [10632.834849] LustreError: 327228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788604190 with bad export cookie 13815628750438556935 [10632.837710] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10632.850553] LustreError: 327228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10633.115984] Lustre: server umount lustre-MDT0001 complete [10641.732922] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [10643.813530] Key type lgssc unregistered [10644.034609] LNet: 330309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10644.045285] LNetError: 330309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10644.060539] LNet: Removed LNI 192.168.203.114@tcp [10644.717757] Key type .llcrypt unregistered [10644.725097] Key type ._llcrypt unregistered [10662.946829] Key type ._llcrypt registered [10662.951432] Key type .llcrypt registered [10663.071771] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [10663.837031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10663.921243] alg: No test for adler32 (adler32-zlib) [10664.955903] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [10665.198579] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [10666.855144] Key type lgssc registered [10667.936435] Lustre: Echo OBD driver; http://www.lustre.org/ [10674.620201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10674.648520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10676.157228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10680.300161] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10685.409421] LustreError: 331579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10685.429442] LustreError: 331579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10687.778659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10688.129271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10692.460054] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10699.474199] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10706.463900] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10714.222492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10714.567854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10720.258142] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10727.224798] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10739.017806] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10740.285026] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10744.288137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10744.297714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10744.320618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10745.049614] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.14@tcp (stopping) [10748.389627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10748.391412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10748.412627] Lustre: Skipped 1 previous similar message [10748.417283] Lustre: Skipped 1 previous similar message [10749.770986] Lustre: server umount lustre-MDT0000 complete [10750.122793] LustreError: 331576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10752.964192] LustreError: 331561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788604310 with bad export cookie 7763265051294596858 [10752.978572] LustreError: 331561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10752.979292] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10753.511528] LustreError: 331574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10753.512611] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10753.551662] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10753.560701] Lustre: Skipped 1 previous similar message [10757.536226] LustreError: 333419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10757.570275] LustreError: 333419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10760.368168] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.14@tcp (stopping) [10760.377935] Lustre: Skipped 2 previous similar messages [10762.656967] LustreError: 331575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10762.674745] LustreError: 331575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10767.327479] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10767.560248] Lustre: server umount lustre-MDT0001 complete [10770.687434] Lustre: server umount lustre-OST0000 complete [10778.047817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10781.905255] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10784.233141] Lustre: 335758:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.114@tcp [10786.053087] Lustre: server umount MGS complete [10793.482452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10793.819273] LustreError: 336331:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10793.846239] LustreError: 336331:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10793.883791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10798.968475] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10807.859961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10808.168859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10813.230679] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10821.896453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10822.337834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10826.928321] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10834.303852] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10835.991589] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [10841.361392] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10842.889368] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10848.229737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10848.242163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10848.267282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10848.277015] Lustre: Skipped 2 previous similar messages [10849.254418] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10853.556934] Lustre: server umount lustre-OST0000 complete [10859.489359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10859.499736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10859.520737] Lustre: Skipped 2 previous similar messages [10864.609053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10864.613602] Lustre: Skipped 5 previous similar messages [10871.264467] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10871.405970] Lustre: server umount lustre-MDT0000 complete [10874.836432] LustreError: 337066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788604432 with bad export cookie 7763265051294597698 [10874.841743] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10874.844016] LustreError: 337066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10875.168628] Lustre: server umount lustre-MDT0001 complete [10885.547501] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [10888.455434] Key type lgssc unregistered [10888.709698] LNet: 339768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10888.717977] LNetError: 339768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10888.737386] LNet: Removed LNI 192.168.203.114@tcp [10889.550000] Key type .llcrypt unregistered [10889.553844] Key type ._llcrypt unregistered [10905.415304] Key type ._llcrypt registered [10905.422529] Key type .llcrypt registered [10905.520321] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [10920.285281] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [10921.365389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10921.486385] alg: No test for adler32 (adler32-zlib) [10922.457409] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [10922.772771] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [10924.463801] Key type lgssc registered [10925.604816] Lustre: Echo OBD driver; http://www.lustre.org/ [10960.252560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10960.286315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10961.632762] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10961.670594] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10961.779226] Lustre: lustre-MDT0000: new disk, initializing [10961.863531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10961.878606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10966.555285] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10975.901035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10976.016599] Lustre: 342925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10976.083324] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10976.093866] Lustre: Skipped 1 previous similar message [10976.167542] Lustre: lustre-MDT0001: new disk, initializing [10976.279901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10976.329169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10976.345051] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10980.542462] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10988.915508] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10993.856491] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11000.112187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11000.402864] Lustre: lustre-OST0000: new disk, initializing [11000.406203] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11000.419562] Lustre: 344893:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11000.511766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11006.206028] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11010.102250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11010.109765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11010.214831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11014.189753] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11018.623737] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11018.836095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11022.330713] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11022.517291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11025.893665] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11025.897072] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11025.905572] LustreError: Skipped 1 previous similar message [11025.909536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11025.917152] Lustre: Skipped 1 previous similar message [11031.010133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11031.020451] Lustre: Skipped 2 previous similar messages [11036.131735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11036.145456] Lustre: Skipped 1 previous similar message [11038.175147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11038.343851] Lustre: server umount lustre-OST0000 complete [11042.785309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11042.795344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11042.813993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11046.368531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11047.961875] Lustre: server umount lustre-MDT0000 complete [11051.011848] LustreError: 343721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788604608 with bad export cookie 12785243607387439777 [11051.015151] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11051.021398] LustreError: 343721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11051.261396] Lustre: server umount lustre-MDT0001 complete [11058.531488] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 06:36:54 (1788604614) [11072.976377] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [11082.310165] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [11116.095520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11116.329671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11116.372122] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11116.473587] Lustre: lustre-MDT0000: new disk, initializing [11116.615248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11116.629632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11120.366681] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11129.616932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11129.726603] Lustre: 349649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11129.770921] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11129.779029] Lustre: Skipped 1 previous similar message [11129.847726] Lustre: lustre-MDT0001: new disk, initializing [11129.924193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11129.955126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11129.967438] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11133.918982] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11141.694968] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11145.594474] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11150.879036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11151.155248] Lustre: lustre-OST0000: new disk, initializing [11151.157948] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11151.165582] Lustre: 351618:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11151.242331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11158.181743] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11161.231791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11161.246169] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11161.351325] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11168.782829] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11197.411189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11197.414919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11197.416916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11197.427906] Lustre: Skipped 1 previous similar message [11197.446897] Lustre: Skipped 3 previous similar messages [11202.888211] Lustre: server umount lustre-OST0000 complete [11206.603233] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11206.620714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11206.642966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11206.651516] Lustre: Skipped 2 previous similar messages [11212.898365] Lustre: server umount lustre-MDT0000 complete [11216.282330] LustreError: 351617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788604774 with bad export cookie 12785243607387440659 [11216.287315] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11216.297563] LustreError: 351617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11216.597888] Lustre: server umount lustre-MDT0001 complete [11228.397498] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [11231.251467] Key type lgssc unregistered [11231.491694] LNet: 353769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11231.504413] LNetError: 353769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11231.531647] LNet: Removed LNI 192.168.203.114@tcp [11232.470299] Key type .llcrypt unregistered [11232.476673] Key type ._llcrypt unregistered [11315.660818] Key type ._llcrypt registered [11315.662780] Key type .llcrypt registered [11315.728591] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [11315.973480] Key type .llcrypt unregistered [11315.979340] Key type ._llcrypt unregistered [11383.994756] Key type ._llcrypt registered [11383.997120] Key type .llcrypt registered [11384.092282] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [11384.357315] Key type .llcrypt unregistered [11384.363665] Key type ._llcrypt unregistered [11412.331566] Key type ._llcrypt registered [11412.334390] Key type .llcrypt registered [11412.435758] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [11412.696385] Key type .llcrypt unregistered [11412.705630] Key type ._llcrypt unregistered [11438.647647] Key type ._llcrypt registered [11438.650202] Key type .llcrypt registered [11438.747711] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [11438.971352] Key type .llcrypt unregistered [11438.973741] Key type ._llcrypt unregistered [11500.888131] Key type ._llcrypt registered [11500.889493] Key type .llcrypt registered [11500.975179] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [11501.178438] Key type .llcrypt unregistered [11501.181063] Key type ._llcrypt unregistered [11523.921320] Key type ._llcrypt registered [11523.923944] Key type .llcrypt registered [11524.001654] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [11524.273032] Key type .llcrypt unregistered [11524.279944] Key type ._llcrypt unregistered [11548.063425] Key type ._llcrypt registered [11548.065036] Key type .llcrypt registered [11548.130736] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [11548.301324] Key type .llcrypt unregistered [11548.303144] Key type ._llcrypt unregistered [11574.569451] Key type ._llcrypt registered [11574.571838] Key type .llcrypt registered [11574.671684] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [11574.987483] Key type .llcrypt unregistered [11574.994294] Key type ._llcrypt unregistered [11604.308913] Key type ._llcrypt registered [11604.315543] Key type .llcrypt registered [11604.491780] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [11604.891419] Key type .llcrypt unregistered [11604.896063] Key type ._llcrypt unregistered [11636.413770] Key type ._llcrypt registered [11636.415810] Key type .llcrypt registered [11636.524318] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [11636.746979] Key type .llcrypt unregistered [11636.749463] Key type ._llcrypt unregistered [11693.531166] Key type ._llcrypt registered [11693.536183] Key type .llcrypt registered [11693.666724] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [11693.886778] Key type .llcrypt unregistered [11693.889934] Key type ._llcrypt unregistered [11717.396917] Key type ._llcrypt registered [11717.398421] Key type .llcrypt registered [11717.480417] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [11717.717383] Key type .llcrypt unregistered [11717.721984] Key type ._llcrypt unregistered [11743.364031] Key type ._llcrypt registered [11743.366856] Key type .llcrypt registered [11743.470413] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [11743.717646] Key type .llcrypt unregistered [11743.723689] Key type ._llcrypt unregistered [11797.819229] Key type ._llcrypt registered [11797.827762] Key type .llcrypt registered [11798.014460] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [11814.474843] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [11815.770884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11815.786384] alg: No test for adler32 (adler32-zlib) [11816.844696] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [11817.070632] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [11818.719186] Key type lgssc registered [11819.777131] Lustre: Echo OBD driver; http://www.lustre.org/ [11853.252152] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11853.290344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11854.507504] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [11854.533417] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [11854.601801] Lustre: tst250fs-MDT0000: new disk, initializing [11854.698599] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11854.716617] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11859.256454] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11869.844846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11869.991563] Lustre: 373889:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11870.084436] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [11870.087888] Lustre: Skipped 1 previous similar message [11870.195894] Lustre: tst250fs-MDT0001: new disk, initializing [11870.305932] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11870.380116] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11870.402117] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11876.106907] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11881.421772] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11888.116400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11888.398344] Lustre: tst250fs-OST0000: new disk, initializing [11888.401504] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [11888.406161] Lustre: 375522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11888.469328] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11894.293853] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11895.432742] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11895.446970] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11895.504114] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11903.365975] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11903.512754] Lustre: tst250fs-OST0001: new disk, initializing [11903.518101] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [11903.525121] Lustre: 376392:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11903.667476] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11905.592630] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11905.606719] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11905.760336] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11910.383011] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11915.562912] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [11917.965127] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [11917.969651] Lustre: Skipped 2 previous similar messages [11936.751414] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11936.770494] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11936.778616] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11936.793309] Lustre: Skipped 3 previous similar messages [11943.188846] Lustre: server umount tst250fs-MDT0000 complete [11946.587804] LustreError: 373880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788605504 with bad export cookie 14783393398142616863 [11946.590788] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11946.599029] LustreError: 373880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11946.891546] Lustre: server umount tst250fs-MDT0001 complete [11960.802208] Lustre: server umount tst250fs-OST0000 complete [11962.336351] Lustre: 372111:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788605504/real 1788605504] req@ffff990e67eaea00 x1875488668713472/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788605520 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11962.376028] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11962.390461] Lustre: Skipped 3 previous similar messages [11964.584487] Lustre: server umount tst250fs-OST0001 complete [11981.890331] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [11990.075784] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [12017.626705] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 06:52:53 (1788605573) [12019.115682] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12033.616603] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [12043.592149] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [12071.664904] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11807 sec ========= 06:53:47 (1788605627) [12073.199216] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:53:49 (1788605629) === [12076.360268] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:53:52 (1788605632) === [12112.611430] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [12115.452509] Key type lgssc unregistered [12115.807934] LNet: 385514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12115.823531] LNetError: 385514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12115.837934] LNet: Removed LNI 192.168.203.114@tcp [12116.958188] Key type .llcrypt unregistered [12116.964075] Key type ._llcrypt unregistered