[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 492952018 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.002391] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.007372] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009022] pid_max: default: 32768 minimum: 301 [ 0.010135] LSM: Security Framework initializing [ 0.011048] Yama: becoming mindful. [ 0.012039] SELinux: Initializing. [ 0.013066] *** VALIDATE selinux *** [ 0.022889] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028039] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029183] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031126] *** VALIDATE tmpfs *** [ 0.032593] *** VALIDATE proc *** [ 0.034257] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.037298] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038159] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040035] Spectre V2 : User space: Vulnerable [ 0.041016] Speculative Store Bypass: Vulnerable [ 0.044213] debug: unmapping init [mem 0xffffffffa2659000-0xffffffffa2660fff] [ 0.046162] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047808] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048026] ... version: 2 [ 0.049012] ... bit width: 48 [ 0.050012] ... generic registers: 4 [ 0.051017] ... value mask: 0000ffffffffffff [ 0.052025] ... max period: 00007fffffffffff [ 0.053019] ... fixed-purpose events: 3 [ 0.054015] ... event mask: 000000070000000f [ 0.055310] rcu: Hierarchical SRCU implementation. [ 0.057584] smp: Bringing up secondary CPUs ... [ 0.058574] x86: Booting SMP configuration: [ 0.059029] .... node #0, CPUs: #1 #2 #3 [ 0.062404] smp: Brought up 1 node, 4 CPUs [ 0.064016] smpboot: Max logical packages: 1 [ 0.065020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146849] node 0 deferred pages initialised in 79ms [ 0.150111] devtmpfs: initialized [ 0.151279] x86/mm: Memory block size: 128MB [ 0.154989] gcov: version magic: 0x41383552 [ 0.158012] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159100] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160304] pinctrl core: initialized pinctrl subsystem [ 0.161189] [ 0.161557] ************************************************************* [ 0.162014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163018] ** ** [ 0.164016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165012] ** ** [ 0.166012] ** This means that this kernel is built to expose internal ** [ 0.167010] ** IOMMU data structures, which may compromise security on ** [ 0.168018] ** your system. ** [ 0.169012] ** ** [ 0.170011] ** If you see this message and you are not debugging the ** [ 0.171010] ** kernel, report this immediately to your vendor! ** [ 0.172013] ** ** [ 0.173013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174015] ************************************************************* [ 0.175647] NET: Registered protocol family 16 [ 0.176447] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.177039] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.178060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.179497] cpuidle: using governor menu [ 0.180933] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.182450] PCI: Using configuration type 1 for base access [ 0.184134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.195175] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.197048] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.201082] cryptd: max_cpu_qlen set to 1000 [ 0.204255] ACPI: Added _OSI(Module Device) [ 0.206014] ACPI: Added _OSI(Processor Device) [ 0.208023] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.209016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.214301] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.220567] ACPI: Interpreter enabled [ 0.222069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.223011] ACPI: Using IOAPIC for interrupt routing [ 0.225213] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.227414] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.237856] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.240049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.242018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.245139] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.250000] acpiphp: Slot [2] registered [ 0.252174] acpiphp: Slot [5] registered [ 0.253147] acpiphp: Slot [6] registered [ 0.255173] acpiphp: Slot [7] registered [ 0.257133] acpiphp: Slot [8] registered [ 0.260160] acpiphp: Slot [9] registered [ 0.261129] acpiphp: Slot [10] registered [ 0.263288] acpiphp: Slot [3] registered [ 0.265136] acpiphp: Slot [4] registered [ 0.267146] acpiphp: Slot [11] registered [ 0.269114] acpiphp: Slot [12] registered [ 0.270139] acpiphp: Slot [13] registered [ 0.272148] acpiphp: Slot [14] registered [ 0.274105] acpiphp: Slot [15] registered [ 0.276187] acpiphp: Slot [16] registered [ 0.278120] acpiphp: Slot [17] registered [ 0.279148] acpiphp: Slot [18] registered [ 0.281140] acpiphp: Slot [19] registered [ 0.283126] acpiphp: Slot [20] registered [ 0.284098] acpiphp: Slot [21] registered [ 0.285110] acpiphp: Slot [22] registered [ 0.287104] acpiphp: Slot [23] registered [ 0.288102] acpiphp: Slot [24] registered [ 0.290215] acpiphp: Slot [25] registered [ 0.291114] acpiphp: Slot [26] registered [ 0.292098] acpiphp: Slot [27] registered [ 0.294103] acpiphp: Slot [28] registered [ 0.295088] acpiphp: Slot [29] registered [ 0.297140] acpiphp: Slot [30] registered [ 0.299107] acpiphp: Slot [31] registered [ 0.300064] PCI host bridge to bus 0000:00 [ 0.301019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.303028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.306042] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.309027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.311022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.314039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.316210] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.320170] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.324225] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.335016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.339975] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.343025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.345030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.349020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.351468] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.353832] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.355050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.358819] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.363017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.381020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.386016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.393227] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.400191] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.410017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.433064] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.443515] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.451018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.458017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.479018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.493313] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.502015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.510013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.524014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.534043] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.541016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.549015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.567018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.577545] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.584024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.591019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.607017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.617356] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.625015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.631016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.644022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.654984] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.656374] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.659386] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.661362] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.663214] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.667030] iommu: Default domain type: Passthrough [ 0.669652] SCSI subsystem initialized [ 0.671169] ACPI: bus type USB registered [ 0.674157] usbcore: registered new interface driver usbfs [ 0.676082] usbcore: registered new interface driver hub [ 0.678104] usbcore: registered new device driver usb [ 0.680396] pps_core: LinuxPPS API ver. 1 registered [ 0.683013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.687077] PTP clock support registered [ 0.690047] EDAC MC: Ver: 3.0.0 [ 0.692126] PCI: Using ACPI for IRQ routing [ 0.693718] NetLabel: Initializing [ 0.695012] NetLabel: domain hash size = 128 [ 0.697012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.699078] NetLabel: unlabeled traffic allowed by default [ 0.702067] vgaarb: loaded [ 0.704184] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.706013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.713493] clocksource: Switched to clocksource kvm-clock [ 0.821239] VFS: Disk quotas dquot_6.6.0 [ 0.822702] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.825456] *** VALIDATE ramfs *** [ 0.826790] *** VALIDATE hugetlbfs *** [ 0.828611] pnp: PnP ACPI init [ 0.831049] pnp: PnP ACPI: found 6 devices [ 0.852393] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.855711] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.857894] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.860632] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.862532] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.864485] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.867247] NET: Registered protocol family 2 [ 0.869678] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.874292] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.878720] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.884868] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.889181] TCP: Hash tables configured (established 65536 bind 65536) [ 0.891638] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.894584] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.896837] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.899515] NET: Registered protocol family 1 [ 0.901495] RPC: Registered named UNIX socket transport module. [ 0.903901] RPC: Registered udp transport module. [ 0.905402] RPC: Registered tcp transport module. [ 0.907222] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.909381] NET: Registered protocol family 44 [ 0.911086] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.913188] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.915269] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.917524] PCI: CLS 0 bytes, default 64 [ 0.919376] Unpacking initramfs... [ 2.327628] debug: unmapping init [mem 0xffff9df23cc54000-0xffff9df23ffbffff] [ 2.332432] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.334966] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.337777] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.851355] Initialise system trusted keyrings [ 2.853074] Key type blacklist registered [ 2.855211] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.864612] zbud: loaded [ 2.867426] *** VALIDATE nfs *** [ 2.868751] *** VALIDATE nfs4 *** [ 2.870167] pstore: using deflate compression [ 2.874000] Platform Keyring initialized [ 2.969929] NET: Registered protocol family 38 [ 2.971989] Key type asymmetric registered [ 2.973542] Asymmetric key parser 'x509' registered [ 2.975342] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.978580] io scheduler mq-deadline registered [ 2.980375] io scheduler kyber registered [ 2.982231] io scheduler bfq registered [ 2.984244] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.987943] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.990927] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.993863] ACPI: Power Button [PWRF] [ 2.999350] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.008099] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.031236] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.042174] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.062150] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.089728] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.118071] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.124332] Non-volatile memory driver v1.3 [ 3.126067] Linux agpgart interface v0.103 [ 3.163840] virtio_blk virtio1: [vda] 149992 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.167661] vda: detected capacity change from 0 to 76795904 [ 3.182441] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.185570] vdb: detected capacity change from 0 to 1073741824 [ 3.206109] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.209556] vdc: detected capacity change from 0 to 2621440000 [ 3.225510] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.228523] vdd: detected capacity change from 0 to 2621440000 [ 3.244076] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.246732] vde: detected capacity change from 0 to 4294967296 [ 3.266602] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.270244] vdf: detected capacity change from 0 to 4294967296 [ 3.278703] libphy: Fixed MDIO Bus: probed [ 3.284862] usbcore: registered new interface driver usbserial_generic [ 3.287729] usbserial: USB Serial support registered for generic [ 3.290598] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.296125] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.298238] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.301292] mousedev: PS/2 mouse device common for all mice [ 3.305339] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.309650] rtc_cmos 00:05: RTC can wake from S4 [ 3.313988] rtc_cmos 00:05: registered as rtc0 [ 3.314509] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.316728] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.321727] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.325258] intel_pstate: CPU model not supported [ 3.332576] hid: raw HID events driver (C) Jiri Kosina [ 3.334381] usbcore: registered new interface driver usbhid [ 3.335784] usbhid: USB HID core driver [ 3.336989] drop_monitor: Initializing network drop monitor service [ 3.339127] Initializing XFRM netlink socket [ 3.340604] NET: Registered protocol family 10 [ 3.343356] Segment Routing with IPv6 [ 3.344315] NET: Registered protocol family 17 [ 3.345885] mpls_gso: MPLS GSO support [ 3.353945] RAS: Correctable Errors collector initialized. [ 3.356342] AVX version of gcm_enc/dec engaged. [ 3.358214] AES CTR mode by8 optimization enabled [ 3.432599] sched_clock: Marking stable (3432567450, 0)->(4361624780, -929057330) [ 3.436648] registered taskstats version 1 [ 3.439399] Loading compiled-in X.509 certificates [ 3.441423] zswap: loaded using pool lzo/zbud [ 3.467829] Key type big_key registered [ 3.480608] Key type encrypted registered [ 3.482857] ima: No TPM chip found, activating TPM-bypass! [ 3.484783] ima: Allocated hash algorithm: sha1 [ 3.486146] ima: No architecture policies found [ 3.487977] evm: Initialising EVM extended attributes: [ 3.489657] evm: security.selinux [ 3.490732] evm: security.ima [ 3.492099] evm: security.capability [ 3.493362] evm: HMAC attrs: 0x1 [ 3.496127] rtc_cmos 00:05: setting system clock to 2026-09-10 14:21:57 UTC (1789050117) [ 3.503302] debug: unmapping init [mem 0xffffffffa3603000-0xffffffffa37fffff] [ 3.506956] debug: unmapping init [mem 0xffffffffa2382000-0xffffffffa2658fff] [ 3.516313] Write protecting the kernel read-only data: 28672k [ 3.520036] debug: unmapping init [mem 0xffffffffa0a03000-0xffffffffa0bfffff] [ 3.523122] debug: unmapping init [mem 0xffffffffa1314000-0xffffffffa13fffff] [ 3.562694] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.570773] systemd[1]: Detected virtualization kvm. [ 3.573052] systemd[1]: Detected architecture x86-64. [ 3.575424] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.604607] systemd[1]: No hostname configured. [ 3.606798] systemd[1]: Set hostname to . [ 3.609230] random: systemd: uninitialized urandom read (16 bytes read) [ 3.612346] systemd[1]: Initializing machine ID from random generator. [ 3.767469] random: systemd: uninitialized urandom read (16 bytes read) [ 3.769968] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.774694] random: systemd: uninitialized urandom read (16 bytes read) [ 3.776746] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.782638] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.550571] device-mapper: uevent: version 1.0.3 [ 4.552919] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.299190] virtio_net virtio0 ens2: renamed from eth0 [ 5.325621] random: fast init done [ 5.365573] scsi host0: ata_piix [ 5.381969] scsi host1: ata_piix [ 5.383527] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.385676] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.889155] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.999523] random: crng init done [ 10.001320] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.597331] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.894942] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.203816] SELinux: Disabled at runtime. [ 12.269256] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.282293] systemd[1]: Detected virtualization kvm. [ 12.284236] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.812000] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.815488] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.822804] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.827384] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.831289] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.839267] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.850531] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ 13.011471] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.292052] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.570652] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.576066] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.724091] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.738719] EDAC sbridge: Ver: 1.1.2 [ 15.429740] Key type dns_resolver registered [ 15.749744] NFS: Registering the id_resolver key type [ 15.751588] Key type id_resolver registered [ 15.753699] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg450-server login: [ 42.630539] libcfs: loading out-of-tree module taints kernel. [ 42.655357] Key type ._llcrypt registered [ 42.656618] Key type .llcrypt registered [ 42.716431] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 52.086902] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 52.838727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.847496] alg: No test for adler32 (adler32-zlib) [ 53.935524] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 54.319732] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 55.967575] Key type lgssc registered [ 58.237710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 76.673223] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 100.030509] hrtimer: interrupt took 8714241 ns [ 125.032363] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 142.519889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 142.595906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 143.891202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 143.990074] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 144.195512] Lustre: lustre-MDT0000: new disk, initializing [ 144.436263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 144.476760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 150.309362] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 167.839493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 168.054334] Lustre: 6485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 168.153345] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 168.170296] Lustre: Skipped 1 previous similar message [ 168.328355] Lustre: lustre-MDT0001: new disk, initializing [ 168.476799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 168.514104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 168.537208] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 174.372443] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.449698] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 193.101940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 193.635537] Lustre: lustre-OST0000: new disk, initializing [ 193.648519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 193.677500] Lustre: 8427:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 193.794349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.741776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 199.753640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 199.846248] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 201.978413] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.916269] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 219.060389] Lustre: lustre-OST0001: new disk, initializing [ 219.064012] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 219.069617] Lustre: 9500:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 219.144512] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 226.767178] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.920323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 228.936829] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 229.002648] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 239.454873] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 246.979962] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 253.937516] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing check_logdir /tmp/testlogs/ [ 260.206762] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing yml_node [ 264.909530] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 267.436131] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 270.125505] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 271.617643] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 10 10:26:24 EDT 2026 [ 291.115701] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 293.643860] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 311.780356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 311.788441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 311.814671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 315.876908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 315.881329] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.910885] Lustre: Skipped 2 previous similar messages [ 317.406481] Lustre: server umount lustre-MDT0000 complete [ 321.321269] LustreError: 6479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050435 with bad export cookie 17501393237262073714 [ 321.339631] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 326.115184] LustreError: 6493:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 326.117929] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.155281] LustreError: 6493:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 326.177087] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 326.182165] Lustre: Skipped 4 previous similar messages [ 327.949956] Lustre: server umount lustre-MDT0001 complete [ 332.563120] Lustre: server umount lustre-OST0000 complete [ 337.308769] Lustre: server umount lustre-OST0001 complete [ 345.636819] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 357.204113] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 398.373664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 398.807738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 398.864510] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 398.986677] Lustre: lustre-MDT0000: new disk, initializing [ 399.096550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 399.121265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 403.698531] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 414.765254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 415.005535] Lustre: 15052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 415.131087] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 415.138627] Lustre: Skipped 1 previous similar message [ 415.298976] Lustre: lustre-MDT0001: new disk, initializing [ 415.467056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 415.514335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 415.534229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 420.921948] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 431.724459] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 438.440658] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 447.738750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 448.177592] Lustre: lustre-OST0000: new disk, initializing [ 448.182791] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 448.190143] Lustre: 17022:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 448.293603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.642491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 449.659835] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 449.822279] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 455.443749] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 465.274443] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 470.349930] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 470.627709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 475.657343] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 476.032396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 480.737708] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 480.745012] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 480.747353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 480.757209] LustreError: Skipped 1 previous similar message [ 480.783109] Lustre: Skipped 2 previous similar messages [ 484.618092] Lustre: server umount lustre-OST0000 complete [ 490.983980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 491.001488] Lustre: Skipped 1 previous similar message [ 491.001905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 491.013303] Lustre: Skipped 2 previous similar messages [ 495.912880] Lustre: server umount lustre-MDT0000 complete [ 496.106836] LustreError: 15062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.126526] LustreError: 15062:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 499.807122] LustreError: 15043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050613 with bad export cookie 17501393237262075247 [ 499.814556] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 499.825821] LustreError: 15043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 500.365579] Lustre: server umount lustre-MDT0001 complete [ 507.547782] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 10:30:20 (1789050620) [ 535.351895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 536.826308] LustreError: 19897:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.911993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 542.180836] LustreError: 19898:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 543.943722] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 546.280304] LustreError: 19897:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 554.534397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 560.549934] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 569.211848] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 575.600512] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 578.293478] Lustre: Failing over lustre-MDT0000 [ 578.749175] Lustre: server umount lustre-MDT0000 complete [ 579.361548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.382062] LustreError: 19897:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 579.392714] LustreError: 19897:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 595.551660] Lustre: 3616:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789050693/real 1789050693] req@ffff9df187970e00 x1875955069128960/t0(0) o400->MGC192.168.204.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789050709 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 595.569947] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 595.578869] LustreError: 19898:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.594414] LustreError: 19898:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 598.192873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 605.856257] LustreError: 3615:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9df1874e4e00 x1875955069131904/t0(0) o250->MGC192.168.204.150@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 606.353946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 606.364602] Lustre: Skipped 1 previous similar message [ 606.419705] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 611.808918] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 611.818893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 611.854581] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 612.116965] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 623.136281] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 629.216780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 629.220901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 629.236412] Lustre: Skipped 2 previous similar messages [ 629.253425] Lustre: Skipped 1 previous similar message [ 634.499660] Lustre: server umount lustre-MDT0000 complete [ 639.460106] LustreError: 21025:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 639.511158] LustreError: 21025:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 639.577083] LustreError: 19878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050753 with bad export cookie 17501393237262076661 [ 639.614044] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.620909] LustreError: 19878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 640.690523] Lustre: server umount lustre-MDT0001 complete [ 652.336755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 659.800995] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 670.582435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 676.086479] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 686.188036] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 694.216291] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 696.952899] Lustre: Failing over lustre-MDT0000 [ 697.150748] Lustre: server umount lustre-MDT0000 complete [ 700.900158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 700.925917] Lustre: Skipped 1 previous similar message [ 706.019033] LustreError: 23965:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 706.059791] LustreError: 23965:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 716.002700] Lustre: 3617:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789050814/real 1789050814] req@ffff9df291fe6300 x1875955069161984/t0(0) o400->MGC192.168.204.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789050830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 716.041628] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 717.871950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 726.508571] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf2e1706c456e0e9f [ 726.524852] Lustre: MGC192.168.204.150@tcp: Connection restored to 0@lo (at 0@lo) [ 726.533563] Lustre: Skipped 1 previous similar message [ 727.035753] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 732.133100] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 732.145103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 732.178686] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 732.874292] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 743.591552] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 747.561874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 747.567121] Lustre: Skipped 2 previous similar messages [ 752.617466] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 753.919766] Lustre: server umount lustre-MDT0000 complete [ 758.028345] LustreError: 23217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050872 with bad export cookie 17501393237262077599 [ 758.039967] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 758.054853] LustreError: 23217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 758.392435] Lustre: server umount lustre-MDT0001 complete [ 769.496374] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 10:34:42 (1789050882) [ 788.487613] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 799.130794] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 849.342259] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 862.409250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 862.618180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 862.646160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 862.749298] Lustre: lustre-MDT0000: new disk, initializing [ 862.866148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 862.876631] Lustre: Skipped 3 previous similar messages [ 862.891256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 867.915480] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 879.277760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 879.405688] Lustre: 29615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 879.726453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 884.461919] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 894.288452] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 900.841712] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 910.064114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 910.565620] Lustre: 31586:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 912.285262] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 912.336995] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 912.440402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 912.462575] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 918.261226] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 930.143371] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 938.974625] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 968.685735] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 968.689134] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 968.690836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 968.690843] Lustre: Skipped 2 previous similar messages [ 968.713323] Lustre: Skipped 3 previous similar messages [ 978.399936] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 978.659902] Lustre: server umount lustre-OST0000 complete [ 997.343354] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 997.878912] Lustre: server umount lustre-MDT0000 complete [ 998.371386] LustreError: 29623:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 998.388541] LustreError: 29623:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 1002.237941] LustreError: 29606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051116 with bad export cookie 17501393237262078005 [ 1002.242736] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1002.250888] LustreError: 29606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1002.815625] Lustre: server umount lustre-MDT0001 complete [ 1016.011761] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 1019.558378] Key type lgssc unregistered [ 1019.972649] LNet: 33568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.006397] LNetError: 33568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1020.028382] LNet: Removed LNI 192.168.204.150@tcp [ 1021.327180] Key type .llcrypt unregistered [ 1021.329118] Key type ._llcrypt unregistered [ 1048.662375] Key type ._llcrypt registered [ 1048.664687] Key type .llcrypt registered [ 1048.757912] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 1049.878627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1049.930802] alg: No test for adler32 (adler32-zlib) [ 1051.089978] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 1051.347270] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 1053.031246] Key type lgssc registered [ 1054.710556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1064.370711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1064.398566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1066.242621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1072.965532] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1076.711893] LustreError: 34980:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1076.756441] LustreError: 34980:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1081.834580] LustreError: 34979:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1084.171483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1084.576715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1088.854037] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1097.743270] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1106.255553] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1115.603851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1116.292344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1125.045883] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1129.587038] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1133.756890] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1136.788589] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1153.672740] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 10:41:05 (1789051265) [ 1155.820170] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1158.060223] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1164.401254] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 10:41:17 (1789051277) [ 1175.993539] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 10:41:28 (1789051288) [ 1177.351904] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1190.825588] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1190.850703] Lustre: Skipped 2 previous similar messages [ 1193.495873] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 10:41:45 (1789051305) [ 1196.289873] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1196.295805] Lustre: Skipped 2 previous similar messages [ 1206.914768] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1206.930373] Lustre: Skipped 4 previous similar messages [ 1223.441532] Lustre: Disabling parameter general.osc.*.max_pages_per_rpc= in log params [ 1223.454541] Lustre: Skipped 5 previous similar messages [ 1226.118660] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 10:42:18 (1789051338) [ 1230.103368] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1234.876961] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1247.264530] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 10:42:39 (1789051359) [ 1255.620378] Lustre: Disabling parameter general.jobid_name=TESTNAME1 in log params [ 1255.631672] Lustre: Skipped 3 previous similar messages [ 1267.675986] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 10:42:59 (1789051379) [ 1283.552933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1283.565921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1283.578490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1285.607469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1285.607985] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1285.617930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1289.779514] Lustre: server umount lustre-OST0000 complete [ 1295.843747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1295.855716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1295.873768] Lustre: Skipped 1 previous similar message [ 1295.893197] Lustre: Skipped 2 previous similar messages [ 1300.966607] LustreError: 35709:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1301.073123] Lustre: server umount lustre-MDT0000 complete [ 1305.956971] LustreError: 37904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051419 with bad export cookie 12240434032938410993 [ 1305.968077] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1305.968469] LustreError: 37904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1306.292577] Lustre: server umount lustre-MDT0001 complete [ 1318.708956] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 1322.880612] Key type lgssc unregistered [ 1323.346113] LNet: 42844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1323.353544] LNetError: 42844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1323.390790] LNet: Removed LNI 192.168.204.150@tcp [ 1324.866130] Key type .llcrypt unregistered [ 1324.871810] Key type ._llcrypt unregistered [ 1345.196903] Key type ._llcrypt registered [ 1345.199341] Key type .llcrypt registered [ 1345.290515] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 1361.272454] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 1362.498647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1362.541182] alg: No test for adler32 (adler32-zlib) [ 1363.635093] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 1363.857441] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 1365.543458] Key type lgssc registered [ 1366.788165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1408.136678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1408.171984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1409.732222] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1409.780475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1409.991775] Lustre: lustre-MDT0000: new disk, initializing [ 1410.196754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1410.269160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1417.375589] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1430.811616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1431.049542] Lustre: 45997:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1431.143904] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1431.152278] Lustre: Skipped 1 previous similar message [ 1431.281519] Lustre: lustre-MDT0001: new disk, initializing [ 1431.435204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1431.505555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1431.528382] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1436.736981] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1446.683963] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1452.301157] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1460.029317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1460.317452] Lustre: lustre-OST0000: new disk, initializing [ 1460.321856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1460.339287] Lustre: 47969:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1460.495883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1465.124205] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1470.062232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1470.077879] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1470.244153] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1473.643157] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1478.455510] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1478.653402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1482.572830] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1482.779644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1485.794989] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1485.807094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1485.807095] LustreError: Skipped 1 previous similar message [ 1485.812543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1485.816524] Lustre: Skipped 1 previous similar message [ 1485.833830] Lustre: Skipped 1 previous similar message [ 1490.920254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1490.930664] Lustre: Skipped 1 previous similar message [ 1491.501366] Lustre: server umount lustre-OST0000 complete [ 1498.081886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1498.093652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1498.114982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1501.156453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1501.158256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1502.633481] Lustre: server umount lustre-MDT0000 complete [ 1507.579256] LustreError: 46795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051621 with bad export cookie 3669824997029591760 [ 1507.580634] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1507.586922] LustreError: 46795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1508.126918] Lustre: server umount lustre-MDT0001 complete [ 1520.283411] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 10:47:12 (1789051632) [ 1527.833733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1528.248775] LustreError: 50428:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1528.266193] LustreError: 50428:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1528.321561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1532.914840] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1533.416310] LustreError: 50429:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1538.536294] LustreError: 50428:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1542.424305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1542.808820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1547.336674] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1556.092275] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1562.643968] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1571.431715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1571.856337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1578.552740] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1586.905813] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1591.706596] LustreError: 52957:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1592.697642] Lustre: Modifying parameter general.timeout=1 in log params [ 1594.494038] Lustre: Modifying parameter general.timeout=3 in log params [ 1594.497807] Lustre: Skipped 1 previous similar message [ 1597.130898] Lustre: Modifying parameter general.timeout=6 in log params [ 1597.141191] Lustre: Skipped 2 previous similar messages [ 1601.869770] Lustre: Modifying parameter general.timeout=11 in log params [ 1601.875677] Lustre: Skipped 4 previous similar messages [ 1610.676026] Lustre: Modifying parameter general.timeout=21 in log params [ 1610.684717] Lustre: Skipped 9 previous similar messages [ 1627.164802] Lustre: Modifying parameter general.timeout=38 in log params [ 1627.168468] Lustre: Skipped 16 previous similar messages [ 1659.901472] Lustre: Modifying parameter general.timeout=71 in log params [ 1659.905715] Lustre: Skipped 32 previous similar messages [ 1729.938549] Lustre: Disabling parameter general.timeout= in log params [ 1729.947189] Lustre: Skipped 58 previous similar messages [ 1733.752770] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 10:50:46 (1789051846) [ 1737.128055] Lustre: 59911:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1739.561136] Lustre: 60023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1741.749760] Lustre: 60121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1746.648146] Lustre: 60319:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1746.655606] Lustre: 60319:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1778.302960] Lustre: 61789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1778.308267] Lustre: 61789:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1778.334669] LustreError: 61789:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1782.635138] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 10:51:35 (1789051895) [ 1796.067290] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1796.095868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1796.109967] Lustre: Skipped 2 previous similar messages [ 1796.125997] Lustre: Skipped 3 previous similar messages [ 1802.143985] Lustre: server umount lustre-MDT0000 complete [ 1806.310717] LustreError: 50429:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1806.346640] LustreError: 50429:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1806.400554] LustreError: 50408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051920 with bad export cookie 3669824997029592642 [ 1806.407905] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1806.410909] LustreError: 50408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1806.791825] Lustre: server umount lustre-MDT0001 complete [ 1821.858525] Lustre: server umount lustre-OST0000 complete [ 1850.844912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1851.324798] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1851.869368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1857.697248] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1868.475966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1868.572655] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1868.603533] Lustre: 64323:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1873.573750] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1881.880272] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1890.454840] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1898.151437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1898.469754] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1898.881747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1898.908623] Lustre: Skipped 1 previous similar message [ 1907.355187] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 1916.008971] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1917.909960] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1917.922700] Lustre: Skipped 23 previous similar messages [ 1917.972327] Lustre: 66888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1926.625151] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1926.636101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1926.665517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1926.673081] Lustre: Skipped 3 previous similar messages [ 1931.081409] Lustre: server umount lustre-OST0000 complete [ 1937.387652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1937.420931] Lustre: Skipped 2 previous similar messages [ 1942.051199] Lustre: server umount lustre-MDT0000 complete [ 1942.503573] LustreError: 64333:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1942.538460] LustreError: 64333:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1946.296413] LustreError: 64315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052060 with bad export cookie 3669824997029595799 [ 1946.299469] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1946.303612] LustreError: 64315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1946.649106] Lustre: server umount lustre-MDT0001 complete [ 1959.051278] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 1962.048702] Key type lgssc unregistered [ 1962.417279] LNet: 68101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1962.426882] LNetError: 68101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1962.445693] LNet: Removed LNI 192.168.204.150@tcp [ 1963.621129] Key type .llcrypt unregistered [ 1963.623417] Key type ._llcrypt unregistered [ 1988.449926] Key type ._llcrypt registered [ 1988.452873] Key type .llcrypt registered [ 1988.607326] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 1989.911297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1989.939567] alg: No test for adler32 (adler32-zlib) [ 1991.063962] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 1991.393250] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 1993.105094] Key type lgssc registered [ 1994.088254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2003.625471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2003.656737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2005.308145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2011.589600] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2015.713822] LustreError: 69511:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2015.747102] LustreError: 69511:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2020.833810] LustreError: 69510:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2024.190703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2025.001749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2030.770805] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2038.368463] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2044.549868] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2053.029716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2053.862666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2061.403634] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2068.896604] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2088.018509] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2096.608428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2096.613910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2096.622201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2100.196230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2100.198889] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2102.593589] Lustre: server umount lustre-MDT0000 complete [ 2105.322976] LustreError: 70242:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.340873] LustreError: 70242:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2106.028941] LustreError: 69491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052220 with bad export cookie 17621515731685084560 [ 2106.033083] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2106.042504] LustreError: 69491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2106.349198] Lustre: server umount lustre-MDT0001 complete [ 2120.105406] Lustre: server umount lustre-OST0000 complete [ 2144.332321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2144.531197] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2144.582192] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2144.893874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2150.162867] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2160.689925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2160.850924] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2160.889860] Lustre: 74407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2161.591738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2167.363314] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2175.229652] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2180.980291] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2187.821047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2188.088687] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2188.514909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2194.799921] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2203.274488] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2204.838382] Lustre: 76983:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2206.621729] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2212.321887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2212.326511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2212.333714] Lustre: Skipped 1 previous similar message [ 2212.339089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2212.344263] Lustre: Skipped 2 previous similar messages [ 2217.965562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2217.975616] Lustre: Skipped 1 previous similar message [ 2218.296585] Lustre: server umount lustre-OST0000 complete [ 2222.561416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2222.572234] LustreError: Skipped 1 previous similar message [ 2222.577816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2222.589897] Lustre: Skipped 1 previous similar message [ 2222.595113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2222.599806] Lustre: Skipped 1 previous similar message [ 2233.316408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2233.323866] Lustre: Skipped 5 previous similar messages [ 2236.383143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2236.655364] Lustre: server umount lustre-MDT0000 complete [ 2238.433114] LustreError: 75246:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.454438] LustreError: 75246:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2240.361893] LustreError: 74399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052354 with bad export cookie 17621515731685085400 [ 2240.365384] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2240.373883] LustreError: 74399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2240.875880] Lustre: server umount lustre-MDT0001 complete [ 2251.938772] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 2254.372842] Key type lgssc unregistered [ 2254.643471] LNet: 78189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2254.650183] LNetError: 78189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2254.669513] LNet: Removed LNI 192.168.204.150@tcp [ 2255.395643] Key type .llcrypt unregistered [ 2255.400405] Key type ._llcrypt unregistered [ 2281.022455] Key type ._llcrypt registered [ 2281.028315] Key type .llcrypt registered [ 2281.192376] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 2282.027989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2282.188163] alg: No test for adler32 (adler32-zlib) [ 2283.244299] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 2283.454403] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 2285.151352] Key type lgssc registered [ 2286.251198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2293.736730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2293.757481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2295.281664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2299.657866] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2305.511271] LustreError: 79599:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2305.528860] LustreError: 79599:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2308.012678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2308.642627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2313.517725] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2321.535714] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2328.118375] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2334.662663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2335.097523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2341.297649] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2349.458962] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2359.444402] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2360.335137] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2362.051261] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2362.058749] Lustre: Skipped 1 previous similar message [ 2364.558826] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2364.566500] Lustre: Skipped 2 previous similar messages [ 2368.626432] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2368.632828] Lustre: Skipped 4 previous similar messages [ 2376.786435] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2376.792711] Lustre: Skipped 9 previous similar messages [ 2393.427553] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2393.435550] Lustre: Skipped 19 previous similar messages [ 2426.418825] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2426.424577] Lustre: Skipped 36 previous similar messages [ 2454.706616] Lustre: 87710:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2457.640140] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 11:02:50 (1789052570) [ 2459.032366] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2461.669560] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 11:02:53 (1789052573) [ 2466.274179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2466.280606] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2466.288516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2466.300253] Lustre: Skipped 1 previous similar message [ 2466.313869] Lustre: Skipped 1 previous similar message [ 2471.396244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2471.413035] Lustre: Skipped 1 previous similar message [ 2471.951400] Lustre: server umount lustre-OST0000 complete [ 2476.528174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.531174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2476.544131] Lustre: Skipped 1 previous similar message [ 2481.633751] LustreError: 79600:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.681900] Lustre: server umount lustre-MDT0000 complete [ 2485.211826] LustreError: 79580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052599 with bad export cookie 17297921954614620423 [ 2485.221870] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2485.229905] LustreError: 79580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2485.588842] Lustre: server umount lustre-MDT0001 complete [ 2495.614387] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 2497.959618] Key type lgssc unregistered [ 2498.258788] LNet: 88962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2498.269815] LNetError: 88962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2498.290605] LNet: Removed LNI 192.168.204.150@tcp [ 2499.052226] Key type .llcrypt unregistered [ 2499.054714] Key type ._llcrypt unregistered [ 2502.676466] Key type ._llcrypt registered [ 2502.678694] Key type .llcrypt registered [ 2502.797820] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2506.602373] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules [ 2507.473932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2507.518401] alg: No test for adler32 (adler32-zlib) [ 2508.652085] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 2508.880390] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 2510.569711] Key type lgssc registered [ 2512.102524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2513.627214] LustreError: 90163:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2533.805234] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 2535.669043] LustreError: 90163:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2542.781865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2542.815339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2544.391701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2548.965571] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2554.852183] LustreError: 91186:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.887528] LustreError: 91186:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2556.468478] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 11:04:29 (1789052669) [ 2559.968982] LustreError: 91187:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2560.541359] LustreError: 91218:0:(osp_object.c:623:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2560.552686] LustreError: 91218:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2560.576092] LustreError: 91218:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 2561.123180] Lustre: server umount lustre-MDT0000 complete [ 2575.888220] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 2578.925910] Key type lgssc unregistered [ 2579.256637] LNet: 92713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2579.275968] LNetError: 92713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2579.308917] LNet: Removed LNI 192.168.204.150@tcp [ 2580.591636] Key type .llcrypt unregistered [ 2580.597191] Key type ._llcrypt unregistered [ 2599.441435] Key type ._llcrypt registered [ 2599.443909] Key type .llcrypt registered [ 2599.564074] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 2600.715550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2600.745046] alg: No test for adler32 (adler32-zlib) [ 2601.857761] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 2602.104414] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 2603.775462] Key type lgssc registered [ 2604.817852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2612.227891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2612.253878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2613.776965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2618.123619] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2623.991726] LustreError: 93927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2624.017486] LustreError: 93927:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2627.022931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2627.942586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2632.704558] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2641.329836] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2648.800992] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2658.298442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2659.044077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2665.547364] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2674.072564] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2721.895682] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 11:07:15 (1789052835) [ 2723.243588] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2724.605149] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 11:07:17 (1789052837) [ 2728.932296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.949350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2728.966862] Lustre: Skipped 2 previous similar messages [ 2728.970136] Lustre: Skipped 1 previous similar message [ 2734.053734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2734.065333] Lustre: Skipped 3 previous similar messages [ 2734.981362] Lustre: server umount lustre-MDT0000 complete [ 2739.177319] LustreError: 94658:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2739.226434] LustreError: 94658:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2739.648427] LustreError: 94669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052853 with bad export cookie 13682609542256202516 [ 2739.660449] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2739.674441] LustreError: 94669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2740.489845] Lustre: server umount lustre-MDT0001 complete [ 2755.615473] Lustre: server umount lustre-OST0000 complete [ 2770.082860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2770.651507] LustreError: 98228:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2770.863552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2776.038980] LustreError: 98229:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2777.382176] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2780.137670] LustreError: 98228:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2786.289139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2786.960737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2790.961162] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2799.276915] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2804.775731] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2822.621057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2822.857709] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2822.866853] LustreError: 98218:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2822.897293] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2822.912895] LustreError: 100314:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2822.922862] LustreError: 100314:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2822.934397] LustreError: 100314:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2822.995962] Lustre: server umount lustre-OST0000 complete [ 2822.998964] LustreError: 100314:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2833.695816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2833.971404] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2833.983391] LustreError: 98217:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2834.023081] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2834.039799] LustreError: 100781:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2834.046859] LustreError: 100781:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2834.051703] LustreError: 100781:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2834.130701] Lustre: server umount lustre-OST0000 complete [ 2834.135337] LustreError: 100781:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2843.719325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2843.883596] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2843.888964] Lustre: Found index 0 for lustre-OST0000, updating log [ 2843.894280] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2843.950491] Lustre: lustre-OST0000: new disk, initializing [ 2843.953818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2843.958830] Lustre: 101315:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2844.199397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2850.478751] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2854.175926] Lustre: Failing over lustre-OST0000 [ 2854.283583] Lustre: server umount lustre-OST0000 complete [ 2858.464343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2858.484692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2858.511170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2861.538085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2861.538454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2866.657840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2866.668607] Lustre: Skipped 2 previous similar messages [ 2872.287192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2872.600578] Lustre: server umount lustre-MDT0000 complete [ 2876.858900] LustreError: 98208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052990 with bad export cookie 13682609542256205267 [ 2876.863149] LustreError: 98208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2876.863415] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2877.488790] Lustre: server umount lustre-MDT0001 complete [ 2888.994135] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 11:10:00 (1789053000) [ 2908.800546] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 2921.925032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2922.458459] LustreError: 103660:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2922.477720] LustreError: 103660:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2922.622726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2928.881266] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2939.060851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2944.674851] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2948.042059] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2954.839440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2955.567702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2955.573853] Lustre: Skipped 1 previous similar message [ 2961.822991] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2972.002407] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2972.152858] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2972.233963] Lustre: lustre-OST0001: new disk, initializing [ 2972.239405] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2972.246239] Lustre: 105977:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2978.587791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2978.607607] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2978.744195] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2982.044709] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 2993.073638] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3001.008878] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3009.519238] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3009.522969] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3009.539341] Lustre: Skipped 1 previous similar message [ 3009.560531] Lustre: Skipped 4 previous similar messages [ 3014.119211] Lustre: server umount lustre-MDT0001 complete [ 3014.642567] LustreError: 103660:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3014.682020] LustreError: 103660:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 3025.978150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3026.098862] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3026.105781] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3026.152183] Lustre: 103650:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3026.833455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3026.840525] Lustre: Skipped 1 previous similar message [ 3027.840045] LustreError: 107527:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3027.861112] LustreError: 107527:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3027.900813] LustreError: 103653:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3028.186852] LustreError: 107542:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.150@tcp: cfg command failed: rc = -17 [ 3028.204464] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3028.204464] [ 3032.094025] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3034.190641] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3049.557645] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 11:12:40 (1789053160) [ 3080.909790] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 3103.080074] Lustre: 109621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3129.490049] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3133.988910] Lustre: 110756:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3231.204195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3231.222798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3231.249927] Lustre: Skipped 1 previous similar message [ 3231.253883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3231.273801] Lustre: Skipped 1 previous similar message [ 3236.683547] Lustre: server umount lustre-MDT0000 complete [ 3236.841329] LustreError: 103656:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3236.888977] LustreError: 103656:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 3241.966778] LustreError: 105976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789053355 with bad export cookie 13682609542256206261 [ 3241.970484] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3241.985543] LustreError: 105976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3242.660490] Lustre: server umount lustre-MDT0001 complete [ 3258.485279] Lustre: server umount lustre-OST0000 complete [ 3273.519421] Lustre: server umount lustre-OST0001 complete [ 3280.222037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3292.083554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3302.303830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3317.306853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3344.045092] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 3357.944843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3357.990781] Lustre: lustre-MDT0000: reset Object Index mappings [ 3358.708888] LustreError: 114708:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3358.727445] LustreError: 114708:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3358.913762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3364.431564] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3378.333767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3378.384240] Lustre: lustre-MDT0001: reset Object Index mappings [ 3385.918631] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3390.909978] Lustre: 115866:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3399.710760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3403.852718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3404.045638] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3408.817576] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3420.002525] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3426.319306] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3426.320234] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3428.807155] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3438.700951] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3442.949177] Lustre: 117749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3508.732464] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 11:20:20 (1789053620) [ 3537.891289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3537.908946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3537.944274] Lustre: Skipped 3 previous similar messages [ 3537.970383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3537.982978] Lustre: Skipped 3 previous similar messages [ 3540.945939] Lustre: server umount lustre-MDT0000 complete [ 3544.051546] LustreError: 117779:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3544.133928] LustreError: 117779:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 3547.059476] LustreError: 114688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789053661 with bad export cookie 13682609542256253112 [ 3547.067671] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3547.090455] LustreError: 114688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3554.028130] Lustre: server umount lustre-MDT0001 complete [ 3565.357085] Lustre: server umount lustre-OST0000 complete [ 3576.775938] Lustre: server umount lustre-OST0001 complete [ 3586.755187] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 3600.235035] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 3647.168824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3647.575700] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3647.624381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3647.716990] Lustre: lustre-MDT0000: new disk, initializing [ 3647.812413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3647.820722] Lustre: Skipped 3 previous similar messages [ 3647.837394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3654.777805] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3663.674770] Lustre: server umount lustre-MDT0000 complete [ 3677.004250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3677.643865] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3677.677841] Lustre: 122024:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3677.694089] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3683.929188] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3690.031636] Lustre: server umount lustre-MDT0000 complete [ 3702.323676] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 11:23:34 (1789053814) [ 3704.922592] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3708.114459] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 11:23:39 (1789053819) [ 3734.170830] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 3748.114567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3754.065136] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3767.175621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3767.416053] Lustre: 124095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3767.527358] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3767.533745] Lustre: Skipped 1 previous similar message [ 3767.710830] Lustre: lustre-MDT0001: new disk, initializing [ 3768.061299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3768.089297] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3774.280815] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3780.853196] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3789.448653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3789.818418] Lustre: lustre-OST0000: new disk, initializing [ 3789.825324] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3789.837210] Lustre: 125672:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3791.775386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3791.793450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3791.943151] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3798.818744] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3812.110894] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3812.334100] Lustre: lustre-OST0001: new disk, initializing [ 3812.339466] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3812.345458] Lustre: 126542:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3813.697460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3813.710231] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3813.937268] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3821.116900] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 3833.660976] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3839.732554] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4332.839896] Lustre: 132970:0:(osd_internal.h:1452:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4332.855162] Lustre: 132970:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4332.860320] Lustre: 132970:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4332.871204] Lustre: 132970:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4332.880905] Lustre: 132970:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4332.890734] Lustre: 132970:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4657.120074] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:39:28 (1789054768) [ 4674.022290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4674.035592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4674.062987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4674.073586] Lustre: Skipped 6 previous similar messages [ 4674.092067] Lustre: Skipped 11 previous similar messages [ 4680.149158] Lustre: server umount lustre-MDT0000 complete [ 4684.258796] LustreError: 124100:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4684.282215] LustreError: 124100:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4684.348621] LustreError: 124086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054798 with bad export cookie 13682609542256324568 [ 4684.363020] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4684.364550] LustreError: 124086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4689.379487] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4689.394897] Lustre: Skipped 2 previous similar messages [ 4689.402123] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4689.411564] Lustre: Skipped 5 previous similar messages [ 4690.903410] Lustre: server umount lustre-MDT0001 complete [ 4701.462943] Lustre: server umount lustre-OST0000 complete [ 4711.886226] Lustre: server umount lustre-OST0001 complete [ 4719.974512] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 4732.026794] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 4778.391047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4778.816250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4778.850232] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4779.010053] Lustre: lustre-MDT0000: new disk, initializing [ 4779.139308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4779.142847] Lustre: Skipped 5 previous similar messages [ 4779.156840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4784.562289] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 4796.752221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4796.870627] Lustre: 136037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4796.918734] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4796.922655] Lustre: Skipped 1 previous similar message [ 4796.982612] Lustre: lustre-MDT0001: new disk, initializing [ 4797.103658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4797.114870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4803.655832] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 4816.434305] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4823.902980] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4833.639033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4834.022362] Lustre: lustre-OST0000: new disk, initializing [ 4834.030285] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4834.047404] Lustre: 138004:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4835.404769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4835.414338] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4835.556342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4841.941173] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 4857.857975] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4874.487705] Lustre: lustre-MDD0000: changelog on [ 4878.970956] Lustre: lustre-MDD0001: changelog on [ 7222.681656] Lustre: lustre-MDD0001: changelog off [ 7224.795181] Lustre: 138786:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 7225.832979] Lustre: lustre-MDD0000: changelog off [ 7237.093221] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7237.101274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7237.102533] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7237.118746] Lustre: Skipped 2 previous similar messages [ 7242.217334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7242.229525] Lustre: Skipped 2 previous similar messages [ 7247.330936] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7247.342799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7247.347418] Lustre: Skipped 1 previous similar message [ 7247.588289] Lustre: server umount lustre-OST0000 complete [ 7252.449959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7252.467074] Lustre: Skipped 1 previous similar message [ 7257.568791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7257.585078] Lustre: Skipped 3 previous similar messages [ 7258.741832] Lustre: server umount lustre-MDT0000 complete [ 7262.692927] LustreError: 136045:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7262.737128] LustreError: 136045:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7263.866244] LustreError: 136030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789057377 with bad export cookie 13682609542257108057 [ 7263.868000] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7263.883755] LustreError: 136030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7264.406652] Lustre: server umount lustre-MDT0001 complete [ 7279.165813] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 7283.236298] Key type lgssc unregistered [ 7283.599160] LNet: 147238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7283.620932] LNetError: 147238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7284.646845] LNet: Removed LNI 192.168.204.150@tcp [ 7285.682199] Key type .llcrypt unregistered [ 7285.687832] Key type ._llcrypt unregistered [ 7310.716769] Key type ._llcrypt registered [ 7310.727355] Key type .llcrypt registered [ 7310.930934] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 7328.267193] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 7329.606803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7329.682279] alg: No test for adler32 (adler32-zlib) [ 7331.000519] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 7331.398552] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 7333.183230] Key type lgssc registered [ 7334.319373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7379.298928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7379.321910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7380.684257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7380.787421] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7380.916923] Lustre: lustre-MDT0000: new disk, initializing [ 7381.116988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7381.289647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7389.251378] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7406.579914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7406.697065] Lustre: 150436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7406.746170] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7406.750447] Lustre: Skipped 1 previous similar message [ 7406.815240] Lustre: lustre-MDT0001: new disk, initializing [ 7406.997454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7407.061592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7407.079731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7413.613851] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7425.760215] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7433.124381] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7441.004194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7441.414917] Lustre: lustre-OST0000: new disk, initializing [ 7441.418701] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7441.423499] Lustre: 152407:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7441.553943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7448.175136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7448.203819] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7448.401428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7449.287796] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7460.354538] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7468.435031] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_module obdecho/obdecho [ 7470.781317] LustreError: 153412:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7470.803128] LustreError: 153412:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7473.189597] LustreError: 153508:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7488.997614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7489.001083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7489.012736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7489.505027] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7489.516172] Lustre: Skipped 1 previous similar message [ 7489.528310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7489.542072] Lustre: Skipped 2 previous similar messages [ 7494.626958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7494.635341] Lustre: Skipped 2 previous similar messages [ 7495.012995] Lustre: server umount lustre-MDT0000 complete [ 7499.676046] LustreError: 151234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789057613 with bad export cookie 7843681454460090884 [ 7499.681833] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7499.695027] LustreError: 151234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7499.749839] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7499.775648] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7500.220787] Lustre: server umount lustre-MDT0001 complete [ 7507.316533] Lustre: server umount lustre-OST0000 complete [ 7532.314745] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 7543.270481] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 7585.155310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7585.415955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7585.449062] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7585.520842] Lustre: lustre-MDT0000: new disk, initializing [ 7585.596098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7585.625164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7591.088554] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7605.317363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7605.468484] Lustre: 157228:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7605.533684] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7605.541190] Lustre: Skipped 1 previous similar message [ 7605.643445] Lustre: lustre-MDT0001: new disk, initializing [ 7605.775878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7605.830287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7605.849923] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7611.851800] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7622.725206] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7629.036896] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7637.625023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7638.022695] Lustre: lustre-OST0000: new disk, initializing [ 7638.024961] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7638.045987] Lustre: 159196:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7638.148175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7645.652151] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7647.351021] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7647.365629] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7647.505630] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7656.386241] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7662.017554] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7662.197545] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7668.089953] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7668.274861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7673.320876] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7673.331995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7673.339275] Lustre: Skipped 1 previous similar message [ 7673.340645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7673.346710] Lustre: Skipped 1 previous similar message [ 7676.851587] Lustre: server umount lustre-OST0000 complete [ 7682.528732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7682.535899] LustreError: Skipped 1 previous similar message [ 7682.540108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7682.554836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7687.599374] Lustre: server umount lustre-MDT0000 complete [ 7688.672653] LustreError: 157235:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7688.708754] LustreError: 157235:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7692.255369] LustreError: 157223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789057806 with bad export cookie 7843681454460091843 [ 7692.259635] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7692.267683] LustreError: 157223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7692.709431] Lustre: server umount lustre-MDT0001 complete [ 7696.802578] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 12:30:09 (1789057809) [ 7698.649793] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7700.652517] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 12:30:13 (1789057813) [ 7710.232164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7710.920771] LustreError: 161610:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7711.020238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7716.329042] LustreError: 161611:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7716.723149] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7720.418411] LustreError: 161610:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7725.544632] LustreError: 161611:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7729.469913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7737.211512] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7746.784713] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7754.517513] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7763.007639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7763.378761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7763.382784] Lustre: Skipped 1 previous similar message [ 7772.512961] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7783.362438] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7792.104525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7792.109773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7792.131678] Lustre: Skipped 1 previous similar message [ 7792.134176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7792.150309] Lustre: Skipped 4 previous similar messages [ 7796.784367] Lustre: server umount lustre-MDT0000 complete [ 7797.220405] LustreError: 161610:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7797.244404] LustreError: 161610:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7801.189176] LustreError: 161590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789057915 with bad export cookie 7843681454460092725 [ 7801.206227] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7801.223502] LustreError: 161590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7807.600393] Lustre: server umount lustre-MDT0001 complete [ 7815.389899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7835.546355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7835.985767] LustreError: 165457:0:(llog.c:1655:llog_backup()) MGC192.168.204.150@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7836.004056] Lustre: 165457:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.150@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7836.128704] LustreError: 165462:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7836.155361] LustreError: 165462:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7871.424692] LustreError: 165462:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7871.450923] LustreError: 165462:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [ 7896.727628] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7896.742877] Lustre: 165457:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.150@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7935.953430] LustreError: 165464:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7935.968333] LustreError: 165464:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 40 previous similar messages [ 7937.536661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7942.174216] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7942.642363] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7951.673279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7967.206922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7971.366114] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6cda5ab9a869da8e [ 7971.381943] Lustre: MGC192.168.204.150@tcp: Connection restored to 0@lo (at 0@lo) [ 7976.951668] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7976.997840] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 7986.325927] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7988.318596] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7994.858265] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7996.874152] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8017.899229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8017.903601] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8017.917259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8017.930812] Lustre: Skipped 4 previous similar messages [ 8017.938601] Lustre: Skipped 5 previous similar messages [ 8022.856416] Lustre: server umount lustre-MDT0000 complete [ 8026.872957] LustreError: 161591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789058140 with bad export cookie 7843681454460099214 [ 8026.888539] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8026.896856] LustreError: 161591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8027.338025] Lustre: server umount lustre-MDT0001 complete [ 8042.930449] Lustre: server umount lustre-OST0000 complete [ 8069.363855] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 8083.623706] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 8127.639638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8127.909914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8127.954778] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8128.073557] Lustre: lustre-MDT0000: new disk, initializing [ 8128.163942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8128.170596] Lustre: Skipped 1 previous similar message [ 8128.186617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8133.943034] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8145.535761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8145.628684] Lustre: 170588:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8145.861168] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8151.070504] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8161.289412] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8167.972522] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8176.132397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8176.372105] Lustre: lustre-OST0000: new disk, initializing [ 8176.375615] Lustre: Skipped 1 previous similar message [ 8176.382507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8176.391170] Lustre: Skipped 2 previous similar messages [ 8176.402657] Lustre: 172557:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8182.898643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8182.918383] Lustre: Skipped 1 previous similar message [ 8182.930341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8183.126745] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8184.711354] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8197.610363] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8204.353310] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8204.616503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8212.630856] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8213.051699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8219.107294] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8219.117861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8219.120374] LustreError: Skipped 1 previous similar message [ 8219.123456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8219.139397] Lustre: Skipped 2 previous similar messages [ 8230.367611] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8230.536450] Lustre: server umount lustre-OST0000 complete [ 8240.953662] Lustre: server umount lustre-MDT0000 complete [ 8244.704299] LustreError: 170593:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8244.728446] LustreError: 170593:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 36 previous similar messages [ 8244.759484] LustreError: 170579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789058358 with bad export cookie 7843681454460116021 [ 8244.767785] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8244.777041] LustreError: 170579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8245.401226] Lustre: server umount lustre-MDT0001 complete [ 8255.869686] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 12:39:28 (1789058368) [ 8264.471468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8270.026429] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8281.004107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8286.916081] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8296.674455] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8304.111067] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8312.159702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8319.658101] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8328.593931] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8350.552278] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 12:41:03 (1789058463) [ 8356.832482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8356.837146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8356.841776] LustreError: Skipped 1 previous similar message [ 8356.853149] Lustre: Skipped 2 previous similar messages [ 8356.853938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8356.867441] Lustre: Skipped 9 previous similar messages [ 8361.911516] Lustre: server umount lustre-OST0000 complete [ 8373.314364] Lustre: server umount lustre-MDT0000 complete [ 8380.071701] LustreError: 174955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789058494 with bad export cookie 7843681454460116896 [ 8380.084461] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8380.094292] LustreError: 174955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8380.817914] Lustre: server umount lustre-MDT0001 complete [ 8394.425851] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 8399.112915] Key type lgssc unregistered [ 8399.436543] LNet: 178727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8399.443474] LNetError: 178727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8400.494414] LNet: Removed LNI 192.168.204.150@tcp [ 8401.422654] Key type .llcrypt unregistered [ 8401.430720] Key type ._llcrypt unregistered [ 8424.913622] Key type ._llcrypt registered [ 8424.916436] Key type .llcrypt registered [ 8425.037856] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 8426.225113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8426.338065] alg: No test for adler32 (adler32-zlib) [ 8427.731625] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 8428.137910] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 8429.880581] Key type lgssc registered [ 8431.399983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8443.849065] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8443.886313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8459.552117] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8464.672206] LustreError: 179968:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.150@tcp: failed processing log, type 4: rc = -110 [ 8469.792921] LustreError: 179968:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8469.807854] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8469.821886] LustreError: MGC192.168.204.150@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8469.844410] LustreError: 179968:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8469.862249] LustreError: 179968:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 8469.874968] LustreError: 179968:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 8470.027254] Lustre: server umount lustre-OST0000 complete [ 8470.029409] LustreError: 179968:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8481.324637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8483.312228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8488.405072] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8493.540921] LustreError: 180467:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8493.573343] LustreError: 180467:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8498.656584] LustreError: 180466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8499.724975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8500.219675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8507.507898] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8520.406744] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8528.724105] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8537.642791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8538.021770] LustreError: 182285:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8538.063194] Lustre: 182285:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.150@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8538.305691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8546.498164] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8557.334426] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8561.632274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8561.647484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8561.689403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8564.195435] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8564.197815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8564.248617] Lustre: Skipped 2 previous similar messages [ 8567.285975] Lustre: server umount lustre-MDT0000 complete [ 8569.319815] LustreError: 180466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8569.362236] LustreError: 180466:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8573.011936] LustreError: 180449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789058687 with bad export cookie 5784879943657201891 [ 8573.026661] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8573.373957] Lustre: server umount lustre-MDT0001 complete [ 8589.613708] Lustre: server umount lustre-OST0000 complete [ 8602.467938] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 8618.810546] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 8668.499534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8668.769503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8668.805685] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8668.904396] Lustre: lustre-MDT0000: new disk, initializing [ 8669.009982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8669.041304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8675.407024] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8688.675156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8688.765658] Lustre: 185836:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8688.848382] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8688.862318] Lustre: Skipped 1 previous similar message [ 8689.030617] Lustre: lustre-MDT0001: new disk, initializing [ 8689.252154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8689.400655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8689.423591] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8695.437300] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8706.900549] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8714.319421] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8723.333436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8723.800953] Lustre: lustre-OST0000: new disk, initializing [ 8723.806238] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8723.813877] Lustre: 187810:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8723.916785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8729.244558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8729.257336] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8729.479623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8732.070779] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8744.623940] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8750.698810] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8750.988067] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8757.420776] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8757.800683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8760.292844] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8760.305401] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8760.324349] Lustre: Skipped 1 previous similar message [ 8760.332946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8760.338425] Lustre: Skipped 1 previous similar message [ 8765.411230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8765.420857] Lustre: Skipped 1 previous similar message [ 8766.424367] Lustre: server umount lustre-OST0000 complete [ 8771.040133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8771.048723] LustreError: Skipped 1 previous similar message [ 8771.054324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8771.073340] Lustre: Skipped 1 previous similar message [ 8771.081687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8775.648747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8780.778201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8780.799254] Lustre: Skipped 3 previous similar messages [ 8785.375710] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8785.631507] Lustre: server umount lustre-MDT0000 complete [ 8785.900945] LustreError: 185848:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8790.421708] LustreError: 185829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789058904 with bad export cookie 5784879943657202612 [ 8790.426477] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8790.431360] LustreError: 185829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8791.039491] Lustre: server umount lustre-MDT0001 complete [ 8803.217979] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 12:48:35 (1789058915) [ 8825.521291] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [ 8829.143950] Key type lgssc unregistered [ 8829.448053] LNet: 190755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8829.465663] LNetError: 190755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8829.483831] LNet: Removed LNI 192.168.204.150@tcp [ 8830.743506] Key type .llcrypt unregistered [ 8830.753307] Key type ._llcrypt unregistered [ 8853.821509] Key type ._llcrypt registered [ 8853.824077] Key type .llcrypt registered [ 8854.006897] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 8855.659247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8855.758656] alg: No test for adler32 (adler32-zlib) [ 8857.292370] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 8857.713753] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [ 8859.455838] Key type lgssc registered [ 8861.548494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8870.991450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8871.026273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8872.604678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8878.892052] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8882.147326] LustreError: 191977:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8882.183161] LustreError: 191977:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8887.273680] LustreError: 191978:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8891.145952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8891.616360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8898.038951] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8907.960829] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8915.324224] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8926.422933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8926.747407] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8926.753267] LustreError: 193843:0:(llog.c:1696:llog_backup()) MGC192.168.204.150@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8926.762182] Lustre: 193843:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.150@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8927.048351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8937.061992] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 8948.056333] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8953.312381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8953.326243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8953.345182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8953.843051] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8953.859454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8953.877598] Lustre: Skipped 1 previous similar message [ 8957.386616] Lustre: server umount lustre-MDT0000 complete [ 8958.972424] LustreError: 192705:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8959.006049] LustreError: 192705:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8962.151542] LustreError: 191958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059076 with bad export cookie 7813799242345489818 [ 8962.163701] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8962.164489] LustreError: 191958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8962.556272] Lustre: server umount lustre-MDT0001 complete [ 8978.155232] Lustre: server umount lustre-OST0000 complete [ 8992.049224] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 9007.247706] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 9058.665346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9059.098731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9059.196850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9059.379727] Lustre: lustre-MDT0000: new disk, initializing [ 9059.473664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9059.499617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9066.442891] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9081.306482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9081.493784] Lustre: 197385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9081.546113] Lustre: *** cfs_fail_loc=131e, val=0*** [ 9081.551026] Lustre: Skipped 1 previous similar message [ 9081.560894] LustreError: 198170:0:(llog.c:1696:llog_backup()) MGC192.168.204.150@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 9081.572016] LustreError: 198170:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 9081.583267] Lustre: 198170:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.150@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 9081.624253] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9081.630430] Lustre: Skipped 1 previous similar message [ 9081.727717] Lustre: lustre-MDT0001: new disk, initializing [ 9081.929432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9082.010792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9082.040788] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9088.332540] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9100.623830] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9108.558582] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9116.776537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9116.979716] Lustre: *** cfs_fail_loc=131e, val=0*** [ 9116.982211] Lustre: Skipped 1 previous similar message [ 9116.984818] LustreError: 199333:0:(llog.c:1696:llog_backup()) MGC192.168.204.150@tcp: failed to backup log lustre-OST0000: rc = -28 [ 9116.991608] LustreError: 199333:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 9116.996848] Lustre: 199333:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.150@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 9117.003853] Lustre: 199333:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 9117.045397] Lustre: lustre-OST0000: new disk, initializing [ 9117.052805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9117.066406] Lustre: 199359:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9117.159294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9123.964462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9123.982546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9124.128980] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9126.001670] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9138.354792] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9144.473751] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9144.759178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9150.295646] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9150.529642] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9155.050309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9155.053951] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9155.064860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9155.065526] Lustre: Skipped 2 previous similar messages [ 9160.163940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9160.176344] Lustre: Skipped 2 previous similar messages [ 9165.289771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9165.306964] Lustre: Skipped 1 previous similar message [ 9167.327876] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9167.493291] Lustre: server umount lustre-OST0000 complete [ 9173.984760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9173.992613] LustreError: Skipped 1 previous similar message [ 9173.997380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9174.012356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9178.807498] Lustre: server umount lustre-MDT0000 complete [ 9180.643320] LustreError: 197397:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9180.669790] LustreError: 197397:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9183.763021] LustreError: 197378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059297 with bad export cookie 7813799242345490532 [ 9183.788903] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9184.458703] Lustre: server umount lustre-MDT0001 complete [ 9195.174938] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 12:55:07 (1789059307) [ 9214.941295] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 9230.069496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9230.837944] LustreError: 202491:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9230.915178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9238.956216] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9240.033564] LustreError: 202492:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9240.063361] LustreError: 202492:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9250.197358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9256.421862] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9260.263166] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9268.720514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9269.258399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9269.265705] Lustre: Skipped 1 previous similar message [ 9278.199113] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9289.304543] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9289.536082] Lustre: lustre-OST0001: new disk, initializing [ 9289.546173] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9289.553495] Lustre: 204802:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9296.068095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9296.084641] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9296.230281] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9298.133192] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9311.813746] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9319.419182] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9322.617365] Lustre: DEBUG MARKER: ADD OST3 [ 9330.075270] loop: module loaded [ 9330.191256] loop0: detected capacity change from 0 to 204800000 [ 9330.566468] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9330.578796] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9330.598580] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9338.832356] Lustre: DEBUG MARKER: STOP OST3 [ 9339.280304] loop0: detected capacity change from 0 to 204800000 [ 9346.741553] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9347.062851] Lustre: lustre-OST0003: new disk, initializing [ 9347.067434] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 9347.082091] Lustre: 206870:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9347.287664] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 9347.299891] Lustre: Skipped 1 previous similar message [ 9348.977152] Lustre: *** cfs_fail_loc=2109, val=2*** [ 9349.418305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 9349.446574] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 9349.504708] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 9350.867151] Lustre: Failing over lustre-OST0003 [ 9350.957309] Lustre: server umount lustre-OST0003 complete [ 9351.010622] LustreError: 206967:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 9353.741144] LustreError: 203975:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9353.768701] LustreError: 203975:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9354.236477] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9354.264337] Lustre: Skipped 2 previous similar messages [ 9357.233463] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9371.134309] Lustre: DEBUG MARKER: START OST3 again [ 9380.327737] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9380.831994] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 9382.301500] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 9382.400075] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 9382.402476] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 9387.997810] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9411.931122] Lustre: Failing over lustre-OST0003 [ 9412.220279] Lustre: server umount lustre-OST0003 complete [ 9413.089908] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9413.094875] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9413.098398] LustreError: Skipped 1 previous similar message [ 9413.102131] LustreError: 204237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9413.102143] LustreError: 204237:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [ 9413.171267] Lustre: Skipped 1 previous similar message [ 9435.106804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9435.115626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9435.129546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9435.136545] Lustre: Skipped 2 previous similar messages [ 9438.640773] Lustre: server umount lustre-MDT0000 complete [ 9443.619887] LustreError: 202473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059557 with bad export cookie 7813799242345491421 [ 9443.655063] LustreError: 202473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9443.663207] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9450.722043] Lustre: server umount lustre-MDT0001 complete [ 9456.674572] Lustre: server umount lustre-OST0000 complete [ 9462.965492] Lustre: server umount lustre-OST0001 complete [ 9472.150040] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 9482.277163] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 9531.509188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9531.812832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9531.870801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9532.049136] Lustre: lustre-MDT0000: new disk, initializing [ 9532.129193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9532.137202] Lustre: Skipped 1 previous similar message [ 9532.148455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9538.776822] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9552.766200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9552.954696] Lustre: 212422:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9553.060332] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9553.064896] Lustre: Skipped 1 previous similar message [ 9553.184337] Lustre: lustre-MDT0001: new disk, initializing [ 9553.410469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9553.421399] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9560.817340] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9572.223842] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9580.100252] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9589.883446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9590.421563] Lustre: lustre-OST0000: new disk, initializing [ 9590.424905] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9590.431482] Lustre: 214403:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9592.226415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9592.254921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9592.417093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9599.035854] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9609.573160] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9615.521621] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9615.776706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9621.612876] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9621.888906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9628.150503] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9628.159351] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9628.170033] Lustre: Skipped 5 previous similar messages [ 9628.188919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9628.207562] LustreError: Skipped 1 previous similar message [ 9628.232195] Lustre: Skipped 9 previous similar messages [ 9631.030501] Lustre: server umount lustre-OST0000 complete [ 9642.793809] Lustre: server umount lustre-MDT0000 complete [ 9643.488529] LustreError: 212433:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9643.517061] LustreError: 212433:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 20 previous similar messages [ 9647.339536] LustreError: 212414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059761 with bad export cookie 7813799242345493710 [ 9647.342132] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9647.347669] LustreError: 212414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9647.825727] Lustre: server umount lustre-MDT0001 complete [ 9652.217889] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 13:02:44 (1789059764) [ 9671.286408] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [ 9685.526280] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [ 9729.801072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9730.032725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9730.065535] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9730.236256] Lustre: lustre-MDT0000: new disk, initializing [ 9730.378914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9735.567867] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9748.807722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9748.941663] Lustre: 219037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9749.267857] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9755.475372] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9767.727687] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9774.396491] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9782.515323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9782.934036] Lustre: 221007:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9784.370888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9784.442876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9790.015830] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9800.897740] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9806.929429] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9807.324061] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9813.974755] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9814.328709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9820.149432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9820.158510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9820.161116] Lustre: Skipped 4 previous similar messages [ 9820.176451] Lustre: Skipped 3 previous similar messages [ 9831.395593] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9831.565288] Lustre: server umount lustre-OST0000 complete [ 9836.514183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9842.649768] Lustre: server umount lustre-MDT0000 complete [ 9845.742608] LustreError: 219045:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9845.769510] LustreError: 219045:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9847.377792] LustreError: 219029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059961 with bad export cookie 7813799242345494599 [ 9847.389096] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9847.401392] LustreError: 219029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9847.981341] Lustre: server umount lustre-MDT0001 complete [ 9859.586226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9860.463113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9860.473790] Lustre: Skipped 5 previous similar messages [ 9865.383347] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9876.451764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9881.860062] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9891.100933] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9899.359436] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9907.848876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9915.082555] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [ 9923.456871] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9946.414275] Lustre: server umount lustre-OST0000 complete [ 9953.760393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9953.776802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9953.799015] Lustre: Skipped 4 previous similar messages [ 9953.806701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9953.812275] Lustre: Skipped 11 previous similar messages [ 9957.720594] Lustre: server umount lustre-MDT0000 complete [ 9962.863239] LustreError: 223207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060076 with bad export cookie 7813799242345495481 [ 9962.877491] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9962.883099] LustreError: 223207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9963.571790] Lustre: server umount lustre-MDT0001 complete [10064.314738] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [10067.861864] Key type lgssc unregistered [10068.303755] LNet: 226909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10068.319334] LNetError: 226909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10068.339811] LNet: Removed LNI 192.168.204.150@tcp [10069.442171] Key type .llcrypt unregistered [10069.443862] Key type ._llcrypt unregistered [10101.118872] Key type ._llcrypt registered [10101.125442] Key type .llcrypt registered [10101.240957] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [10118.237448] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [10119.757079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10119.806269] alg: No test for adler32 (adler32-zlib) [10120.939729] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [10121.192194] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [10122.855937] Key type lgssc registered [10124.227508] Lustre: Echo OBD driver; http://www.lustre.org/ [10162.774134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10162.788818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10164.030213] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10164.073617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10164.201334] Lustre: lustre-MDT0000: new disk, initializing [10164.334108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10164.367343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10170.004849] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10182.415241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10182.625445] Lustre: 230259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10182.707889] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10182.713038] Lustre: Skipped 1 previous similar message [10182.803590] Lustre: lustre-MDT0001: new disk, initializing [10182.931495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10183.060811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10183.076298] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10188.100722] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10199.883180] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10207.555561] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10215.907366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10216.238536] Lustre: lustre-OST0000: new disk, initializing [10216.246299] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10216.258545] Lustre: 232228:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10216.363865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10224.562981] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10226.249357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10226.273970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10226.379620] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10237.448107] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10244.415222] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10244.916728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10251.799555] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10252.023165] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10257.379597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10257.392921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10257.403037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10257.411117] Lustre: Skipped 1 previous similar message [10261.115857] Lustre: server umount lustre-OST0000 complete [10267.631901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10267.649538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10267.668140] Lustre: Skipped 2 previous similar messages [10267.685123] Lustre: Skipped 1 previous similar message [10272.735391] Lustre: server umount lustre-MDT0000 complete [10277.648027] LustreError: 231054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060391 with bad export cookie 16165453061072080970 [10277.653084] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10277.659042] LustreError: 231054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10278.212821] Lustre: server umount lustre-MDT0001 complete [10304.230753] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [10317.235464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10317.828372] LustreError: 235186:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10317.851694] LustreError: 235186:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10317.901119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10322.912155] LustreError: 235187:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10323.311261] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10327.012637] LustreError: 235186:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10332.134055] LustreError: 235187:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10334.236345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10334.853219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10340.793192] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10344.214197] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10352.720971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10353.199128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10360.709631] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10372.723878] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10372.995408] Lustre: lustre-OST0001: new disk, initializing [10372.998675] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10373.004643] Lustre: 237494:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10373.076739] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10382.348716] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10382.533049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10382.548413] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10382.799082] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10396.573959] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10407.677742] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10425.554546] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 13:15:37 (1789060537) [10428.917718] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10428.922647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10428.925133] Lustre: Skipped 2 previous similar messages [10428.958713] Lustre: Skipped 3 previous similar messages [10434.019526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10434.042961] Lustre: Skipped 3 previous similar messages [10439.142974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10439.152595] Lustre: Skipped 2 previous similar messages [10442.207144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10442.649145] Lustre: server umount lustre-MDT0000 complete [10444.260538] LustreError: 235186:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10444.307308] LustreError: 235186:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10447.899872] LustreError: 235167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060561 with bad export cookie 16165453061072081852 [10447.929767] LustreError: 235167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10447.930354] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10448.768649] Lustre: server umount lustre-MDT0001 complete [10463.401584] Lustre: server umount lustre-OST0000 complete [10465.586537] Lustre: 228485:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789060563/real 1789060563] req@ffff9df28555e300 x1875965625101568/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789060579 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10465.629435] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10465.639471] Lustre: Skipped 1 previous similar message [10467.978490] Lustre: server umount lustre-OST0001 complete [10476.135638] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [10489.777073] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [10530.718731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10530.935602] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10530.963980] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10531.064905] Lustre: lustre-MDT0000: new disk, initializing [10531.198840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10531.236541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10535.344742] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10546.980307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10547.130289] Lustre: 241445:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10547.237238] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10547.245524] Lustre: Skipped 1 previous similar message [10547.369088] Lustre: lustre-MDT0001: new disk, initializing [10547.510346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10547.529230] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10552.181586] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10563.175608] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10569.458899] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10577.321137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10577.789653] Lustre: lustre-OST0000: new disk, initializing [10577.812832] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10577.824165] Lustre: 243411:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10579.410968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10579.426419] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10579.602595] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10585.258647] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10596.747686] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10601.735063] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10602.021482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10607.203269] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10607.379500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10610.148992] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10610.163028] LustreError: Skipped 1 previous similar message [10610.167425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10610.182381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10610.188229] Lustre: Skipped 2 previous similar messages [10616.377892] Lustre: server umount lustre-OST0000 complete [10624.484877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10624.491509] LustreError: Skipped 1 previous similar message [10624.499105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10624.524493] Lustre: Skipped 1 previous similar message [10630.624518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10630.635100] Lustre: Skipped 6 previous similar messages [10635.232972] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10635.567030] Lustre: server umount lustre-MDT0000 complete [10635.745362] LustreError: 241453:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10635.757892] LustreError: 241453:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10640.478016] LustreError: 241439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060754 with bad export cookie 16165453061072083987 [10640.485343] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10640.504594] LustreError: 241439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10640.981085] Lustre: server umount lustre-MDT0001 complete [10651.936904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10652.398782] LustreError: 245632:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10652.515565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10652.522450] Lustre: Skipped 2 previous similar messages [10657.612344] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10669.109483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10674.415879] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10683.307519] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10691.052393] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10699.579293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10708.515844] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [10718.113389] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10866.146268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10866.154464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10866.159308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10866.196879] Lustre: Skipped 2 previous similar messages [10870.480304] Lustre: server umount lustre-OST0000 complete [10874.336527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10880.328730] Lustre: server umount lustre-MDT0000 complete [10881.505277] LustreError: 245629:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10881.525199] LustreError: 245629:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10883.784337] LustreError: 246367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060997 with bad export cookie 16165453061072084869 [10883.785949] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10883.792510] LustreError: 246367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10884.213139] Lustre: server umount lustre-MDT0001 complete [10988.982317] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [10991.705261] Key type lgssc unregistered [10991.938551] LNet: 249243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10991.949143] LNetError: 249243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10991.969738] LNet: Removed LNI 192.168.204.150@tcp [10992.748158] Key type .llcrypt unregistered [10992.754970] Key type ._llcrypt unregistered [11019.719711] Key type ._llcrypt registered [11019.723071] Key type .llcrypt registered [11019.842478] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [11039.507814] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [11041.338348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11041.369689] alg: No test for adler32 (adler32-zlib) [11042.636576] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [11043.008431] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [11044.767295] Key type lgssc registered [11046.426772] Lustre: Echo OBD driver; http://www.lustre.org/ [11101.686983] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [11115.262902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11115.329674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11116.795181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11116.854485] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11117.028478] Lustre: lustre-MDT0000: new disk, initializing [11117.152167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11117.186392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11122.877000] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11134.856995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11134.982189] Lustre: 253333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11135.040648] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11135.044713] Lustre: Skipped 1 previous similar message [11135.113844] Lustre: lustre-MDT0001: new disk, initializing [11135.221126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11135.268252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11135.282633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11139.995055] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11145.417450] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11153.911246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11154.153465] Lustre: lustre-OST0000: new disk, initializing [11154.156987] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11154.161962] Lustre: 254969:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11154.239074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11162.258902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11162.272178] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11162.470466] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11163.177278] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11174.914809] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11175.032412] Lustre: lustre-OST0001: new disk, initializing [11175.036680] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11175.046463] Lustre: 255840:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11175.137347] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11180.638691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11180.656836] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11180.763374] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11182.284426] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11195.227090] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11203.233435] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11216.879522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11216.891263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11216.893231] Lustre: Skipped 1 previous similar message [11221.803498] Lustre: server umount lustre-MDT0000 complete [11225.875849] LustreError: 253325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789061339 with bad export cookie 966551651240324727 [11225.887899] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11225.898981] LustreError: 253325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11226.608248] Lustre: server umount lustre-MDT0001 complete [11241.145591] Lustre: server umount lustre-OST0000 complete [11243.942563] Lustre: 250817:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789061341/real 1789061341] req@ffff9df18487b100 x1875966590793600/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789061357 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11243.992814] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11244.032994] Lustre: Skipped 2 previous similar messages [11245.949838] Lustre: server umount lustre-OST0001 complete [11263.022804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11274.092586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11283.806475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [11295.832034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [11317.396343] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [11328.669879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11328.698593] Lustre: lustre-MDT0000: reset Object Index mappings [11329.286739] LustreError: 260853:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11329.300282] LustreError: 260853:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11329.374138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11334.634466] LustreError: 260854:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11334.808515] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11339.746422] LustreError: 260853:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11344.881115] LustreError: 260854:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11345.095833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11345.137177] Lustre: lustre-MDT0001: reset Object Index mappings [11345.516208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11350.807700] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11354.253883] Lustre: 261995:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11363.053411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11363.677939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11370.811433] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11370.868869] LustreError: 262351:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11374.991674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [11380.196406] LustreError: 262350:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11380.222707] LustreError: 262350:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11380.763448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11386.362397] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [11388.651697] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11397.420960] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11401.790932] Lustre: 263868:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11411.940373] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11411.954365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11411.961889] Lustre: Skipped 3 previous similar messages [11411.965078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11411.980077] Lustre: Skipped 6 previous similar messages [11414.911876] Lustre: server umount lustre-MDT0000 complete [11417.060590] LustreError: 262614:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11417.081692] LustreError: 262614:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11418.723332] LustreError: 260833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789061532 with bad export cookie 966551651240331517 [11418.734417] LustreError: 260833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11418.744962] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11419.131470] Lustre: server umount lustre-MDT0001 complete [11434.531610] Lustre: server umount lustre-OST0000 complete [11438.560883] Lustre: 250817:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789061536/real 1789061536] req@ffff9df2aff64e00 x1875966590847872/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789061552 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11438.593749] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11439.377540] Lustre: server umount lustre-OST0001 complete [11462.472993] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 13:32:54 (1789061574) [11481.695626] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [11493.175168] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [11540.859332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11541.192970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11541.255826] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11541.414617] Lustre: lustre-MDT0000: new disk, initializing [11541.639854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11541.649481] Lustre: Skipped 1 previous similar message [11541.684558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11547.582704] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11561.642831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11562.006169] Lustre: 268125:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11562.094769] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11562.100728] Lustre: Skipped 1 previous similar message [11562.185426] Lustre: lustre-MDT0001: new disk, initializing [11562.442898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11562.450856] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11568.629883] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11580.086884] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11587.129599] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11595.367450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11595.923410] Lustre: lustre-OST0000: new disk, initializing [11595.926924] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11595.949915] Lustre: 270095:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11597.058872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11597.080615] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11597.258275] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11603.191045] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11613.312312] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11619.214581] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11619.515651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11625.899414] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11626.222651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11633.125033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11633.125985] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11633.137380] LustreError: Skipped 1 previous similar message [11633.144177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11633.150484] Lustre: Skipped 1 previous similar message [11634.893290] Lustre: server umount lustre-OST0000 complete [11639.265277] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11639.277044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11639.285627] Lustre: Skipped 1 previous similar message [11643.361460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11643.366286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11643.387874] Lustre: Skipped 1 previous similar message [11645.320202] Lustre: server umount lustre-MDT0000 complete [11648.484093] LustreError: 268137:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11648.503287] LustreError: 268137:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11649.820903] LustreError: 270094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789061763 with bad export cookie 966551651240333610 [11649.832707] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11650.218878] Lustre: server umount lustre-MDT0001 complete [11672.816757] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [11684.878365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11685.433544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11685.441600] Lustre: Skipped 2 previous similar messages [11690.467751] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11699.537792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11704.946798] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11708.729859] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11717.112284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11726.349682] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11736.893895] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11737.250416] Lustre: lustre-OST0001: new disk, initializing [11737.256468] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11737.270973] Lustre: 275341:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11739.183466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11739.197661] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11739.341332] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11745.768356] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11760.083298] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11765.627733] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11773.247374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11773.384222] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11785.706499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11785.708545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11785.722695] Lustre: Skipped 2 previous similar messages [11785.742994] Lustre: Skipped 4 previous similar messages [11790.820564] LustreError: 273033:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11790.843717] LustreError: 273033:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [11790.921318] Lustre: server umount lustre-MDT0000 complete [11795.567507] LustreError: 273013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789061909 with bad export cookie 966551651240334485 [11795.580443] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11795.591884] LustreError: 273013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11796.004760] Lustre: server umount lustre-MDT0001 complete [11811.170815] Lustre: server umount lustre-OST0000 complete [11811.281366] Lustre: 250818:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789061909/real 1789061909] req@ffff9df18c74d880 x1875966590972288/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789061925 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11816.539482] Lustre: server umount lustre-OST0001 complete [11837.957826] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [11851.504052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11852.157041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11852.164228] Lustre: Skipped 3 previous similar messages [11857.149674] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11867.011849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11873.937570] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11877.428868] Lustre: 279831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11885.814448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11893.677329] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11897.616946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11904.847661] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11910.669275] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11913.734724] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [11922.971786] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11928.698240] Lustre: 281704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11936.512741] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11936.772423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11958.122325] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 13:41:09 (1789062069) [11975.143126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11975.152080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11975.170230] Lustre: Skipped 2 previous similar messages [11975.188989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11980.541499] Lustre: server umount lustre-MDT0000 complete [11982.312930] LustreError: 278688:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11982.352820] LustreError: 278688:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [11985.049391] LustreError: 278668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789062099 with bad export cookie 966551651240341387 [11985.064886] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11991.892327] Lustre: server umount lustre-MDT0001 complete [12002.925216] Lustre: server umount lustre-OST0000 complete [12007.360720] Lustre: server umount lustre-OST0001 complete [12015.517347] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [12028.171234] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [12087.248841] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [12099.349029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12099.656798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12099.690879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12099.783283] Lustre: lustre-MDT0000: new disk, initializing [12099.903813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12106.257235] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12119.408513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12119.484112] Lustre: 286262:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12119.750439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12119.755642] Lustre: Skipped 4 previous similar messages [12119.798951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12125.036372] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12138.479043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12138.843583] Lustre: 287895:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12140.175589] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12146.054097] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12157.662901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12157.861052] Lustre: 288766:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12159.481253] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12159.495901] Lustre: Skipped 1 previous similar message [12159.654268] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12165.541969] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12176.170273] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12180.905410] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12180.912136] Lustre: Skipped 1 previous similar message [12200.554080] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 13:45:12 (1789062312) [12204.964420] Lustre: server umount lustre-MDT0000 complete [12205.538808] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12205.565599] Lustre: Skipped 6 previous similar messages [12209.116219] LustreError: 287066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789062323 with bad export cookie 966551651240348933 [12209.125986] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12209.127319] LustreError: 287066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [12209.460395] Lustre: server umount lustre-MDT0001 complete [12225.223693] Lustre: server umount lustre-OST0000 complete [12226.018348] Lustre: 250819:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789062324/real 1789062324] req@ffff9df291486300 x1875966591137920/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789062340 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12230.882302] Lustre: 250818:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789062328/real 1789062328] req@ffff9df18b21aa00 x1875966591138176/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789062344 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12232.407114] Lustre: server umount lustre-OST0001 complete [12246.847910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12247.313502] LustreError: 291012:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12247.333561] LustreError: 291012:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [12252.889838] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12263.097802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12268.756539] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12277.264739] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12284.211551] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12291.757705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12299.095389] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12308.084778] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12329.895144] loop1: detected capacity change from 0 to 204800000 [12330.199691] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12330.212570] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12330.226526] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12336.634398] loop1: detected capacity change from 0 to 204800000 [12343.561723] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12343.652428] LustreError: 291003:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs17156: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [12343.669644] LustreError: 291003:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs17156 [12343.676028] LustreError: 291003:0:(mgs_handler.c:517:mgs_target_reg()) fs17156-MDT0001: Error checking target index: rc = -13 [12343.682493] LustreError: fs17156-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12343.695328] LustreError: 294737:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [12343.702615] LustreError: 294737:0:(tgt_mount.c:2128:server_put_super()) no obd fs17156-MDT0001 [12343.711688] LustreError: 294737:0:(tgt_mount.c:116:server_deregister_mount()) fs17156-MDT0001 not registered [12343.742318] Lustre: server umount fs17156-MDT0001 complete [12343.745191] LustreError: 294737:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12353.454031] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12353.642993] Lustre: Setting parameter fs17156-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs17156-MDT0001 [12353.758294] Lustre: srv-fs17156-MDT0001: No data found on store. Initialize space. [12353.772573] Lustre: Skipped 4 previous similar messages [12353.900263] Lustre: fs17156-MDT0001: new disk, initializing [12353.904482] Lustre: Skipped 3 previous similar messages [12354.031677] Lustre: srv-fs17156-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12354.039343] Lustre: 295204:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17156-MDT0001-mdtlov: can't create new llog: rc = -115 [12354.047174] LustreError: 295204:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17156-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12359.060347] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12366.007153] Lustre: server umount fs17156-MDT0001 complete [12375.536213] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12381.331502] loop1: detected capacity change from 0 to 204800000 [12387.423268] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12387.823248] Lustre: 296772:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17156-MDT0001-mdtlov: can't create new llog: rc = -115 [12387.834464] LustreError: 296772:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17156-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12390.894489] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12393.392763] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12399.523817] Lustre: server umount fs17156-MDT0001 complete [12416.492333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12416.499173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12416.511444] Lustre: Skipped 3 previous similar messages [12416.523363] Lustre: Skipped 11 previous similar messages [12419.448821] Lustre: server umount lustre-MDT0000 complete [12429.244581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12429.430978] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12434.594977] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12434.933816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [12444.553741] loop1: detected capacity change from 0 to 204800000 [12451.777896] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12452.115191] Lustre: 299218:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17156-MDT0001-mdtlov: can't create new llog: rc = -115 [12452.124265] LustreError: 299218:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17156-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12456.845880] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12463.794943] Lustre: server umount fs17156-MDT0001 complete [12493.293696] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12493.315778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12498.457134] Lustre: server umount lustre-OST0000 complete [12509.725859] Lustre: server umount lustre-MDT0000 complete [12514.222202] LustreError: 290995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789062628 with bad export cookie 966551651240351614 [12514.234608] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12514.606369] Lustre: server umount lustre-MDT0001 complete [12527.834956] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [12530.983390] Key type lgssc unregistered [12531.280770] LNet: 301563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12531.287951] LNetError: 301563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12531.312180] LNet: Removed LNI 192.168.204.150@tcp [12532.364184] Key type .llcrypt unregistered [12532.366921] Key type ._llcrypt unregistered [12556.942738] Key type ._llcrypt registered [12556.944820] Key type .llcrypt registered [12557.029532] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [12558.145754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12558.312978] alg: No test for adler32 (adler32-zlib) [12559.522900] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [12559.872331] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [12561.545303] Key type lgssc registered [12562.907885] Lustre: Echo OBD driver; http://www.lustre.org/ [12572.237581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12572.264337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12573.886934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12579.099690] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12583.401444] LustreError: 302833:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12583.426771] LustreError: 302833:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12588.519729] LustreError: 302832:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12589.413777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12589.815521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12595.162552] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12604.355508] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12611.506128] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12620.107045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12620.578569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12629.147432] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12634.937449] LustreError: 305248:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12638.283963] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12640.233888] LustreError: 304666:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12644.326445] LustreError: 304664:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12644.350535] LustreError: 304664:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12653.549276] LustreError: 305248:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12653.568617] LustreError: 305248:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12663.430775] loop1: detected capacity change from 0 to 204800000 [12663.749873] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12663.769727] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12663.815671] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12669.923894] LustreError: 304666:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12669.956639] LustreError: 304666:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [12670.239673] loop1: detected capacity change from 0 to 204800000 [12676.465519] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12676.663223] Lustre: fs28818-OST0000: new disk, initializing [12676.666634] Lustre: srv-fs28818-OST0000: No data found on store. Initialize space. [12676.673688] Lustre: 306574:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12676.729659] Lustre: fs28818-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12683.625635] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12692.633809] Lustre: server umount fs28818-OST0000 complete [12704.741152] LustreError: 304664:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12704.759646] LustreError: 304664:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [12706.736419] loop1: detected capacity change from 0 to 204800000 [12712.637388] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12712.921349] Lustre: fs28818-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12714.989337] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12720.115238] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12720.417651] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12732.256812] loop2: detected capacity change from 0 to 204800000 [12732.500078] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12732.527231] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12732.545942] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12740.197553] loop2: detected capacity change from 0 to 204800000 [12746.959204] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12747.024347] LustreError: fs28818-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12747.034205] LustreError: fs28818-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12747.040137] LustreError: 309618:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [12747.045397] LustreError: 309618:0:(tgt_mount.c:2128:server_put_super()) no obd fs28818-OST0001 [12747.054612] LustreError: 309618:0:(tgt_mount.c:116:server_deregister_mount()) fs28818-OST0001 not registered [12747.088414] Lustre: server umount fs28818-OST0001 complete [12747.090239] LustreError: 309618:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12760.248129] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12760.496508] Lustre: fs28818-OST0001: new disk, initializing [12760.504802] Lustre: srv-fs28818-OST0001: No data found on store. Initialize space. [12760.526497] Lustre: 310170:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12760.613183] Lustre: fs28818-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12769.914368] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12771.316543] LustreError: 304665:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12771.352173] LustreError: 304665:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 37 previous similar messages [12782.243860] Lustre: server umount fs28818-OST0000 complete [12795.756563] Lustre: server umount fs28818-OST0001 complete [12811.024039] loop1: detected capacity change from 0 to 204800000 [12818.646341] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12819.242425] Lustre: fs28818-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12829.496017] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12841.257745] loop2: detected capacity change from 0 to 204800000 [12850.809836] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12860.355292] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [12868.642629] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12873.707467] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12874.913815] Lustre: server umount fs28818-OST0000 complete [12888.012486] Lustre: server umount fs28818-OST0001 complete [12899.318582] LustreError: 304666:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12899.386775] LustreError: 304666:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 74 previous similar messages [12911.592749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12911.605616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12911.646081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12914.663261] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12914.680376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12919.784567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12919.797542] Lustre: Skipped 1 previous similar message [12924.383175] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12924.641131] Lustre: server umount lustre-OST0000 complete [12930.017825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12930.036959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12930.038143] Lustre: Skipped 1 previous similar message [12930.071574] Lustre: Skipped 1 previous similar message [12935.094412] Lustre: server umount lustre-MDT0000 complete [12939.317296] LustreError: 303576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789063053 with bad export cookie 4250108373453192106 [12939.325797] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12939.331677] LustreError: 303576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12939.731519] Lustre: server umount lustre-MDT0001 complete [12954.309496] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [12957.311499] Key type lgssc unregistered [12957.679148] LNet: 316100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12957.684620] LNetError: 316100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12957.700071] LNet: Removed LNI 192.168.204.150@tcp [12959.083079] Key type .llcrypt unregistered [12959.093883] Key type ._llcrypt unregistered [13004.828982] Key type ._llcrypt registered [13004.836608] Key type .llcrypt registered [13004.991047] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [13024.185272] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [13025.333462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13025.557195] alg: No test for adler32 (adler32-zlib) [13026.683186] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [13027.059631] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [13028.871202] Key type lgssc registered [13030.955526] Lustre: Echo OBD driver; http://www.lustre.org/ [13084.804300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13084.831550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13086.208725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13086.290662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13086.498267] Lustre: lustre-MDT0000: new disk, initializing [13086.608983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13086.635518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13091.513807] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13103.670481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13103.792284] Lustre: 319963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13103.913884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13103.922037] Lustre: Skipped 1 previous similar message [13104.064334] Lustre: lustre-MDT0001: new disk, initializing [13104.206293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13104.273647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13104.281163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13108.689418] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13119.819875] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13127.015884] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13135.903603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13136.235282] Lustre: lustre-OST0000: new disk, initializing [13136.242899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13136.256404] Lustre: 321930:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13136.400402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13143.318535] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13144.169973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13144.185852] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13144.334280] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13155.027919] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13157.448398] Lustre: Mounted lustre-client - version 2.17.58_39_g3ed9deb [13158.869429] Lustre: Unmounted lustre-client [13172.709618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13172.737849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13172.754878] Lustre: Skipped 1 previous similar message [13176.761195] Lustre: server umount lustre-OST0000 complete [13182.954177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13182.964307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13182.972306] Lustre: Skipped 2 previous similar messages [13182.986725] Lustre: Skipped 1 previous similar message [13188.354292] Lustre: server umount lustre-MDT0000 complete [13193.387963] LustreError: 319955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789063307 with bad export cookie 17070777811922247416 [13193.396595] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13193.401868] LustreError: 319955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13193.930249] Lustre: server umount lustre-MDT0001 complete [13208.138522] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [13209.567492] LNet: 1 local NIs in recovery (showing 1): 192.168.204.150@tcp [13211.416978] Key type lgssc unregistered [13211.829831] LNet: 324100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13211.837380] LNetError: 324100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13288.681682] LNet: Removed LNI 192.168.204.150@tcp [13290.004276] Key type .llcrypt unregistered [13290.007594] Key type ._llcrypt unregistered [13315.376972] Key type ._llcrypt registered [13315.385749] Key type .llcrypt registered [13315.552687] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [13337.166467] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [13338.192650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13338.212106] alg: No test for adler32 (adler32-zlib) [13339.520950] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [13339.709923] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [13341.407291] Key type lgssc registered [13342.826552] Lustre: Echo OBD driver; http://www.lustre.org/ [13381.333378] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 14:04:53 (1789063493) [13391.619339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13391.690297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13393.073557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13393.095837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13393.167217] Lustre: lustre-MDT0000: new disk, initializing [13393.219672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13393.232500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13399.519386] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13415.478884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13415.698882] Lustre: 327323:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13415.796345] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13415.808394] Lustre: Skipped 1 previous similar message [13416.014828] Lustre: lustre-MDT0001: new disk, initializing [13416.238977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13416.300474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13416.319337] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13422.698511] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13434.517483] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13443.476432] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13454.285590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13454.797952] Lustre: lustre-OST0000: new disk, initializing [13454.806429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13454.822744] Lustre: 329293:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13454.954487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13463.849229] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13464.659439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13464.675738] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13464.868709] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13475.633635] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13495.798687] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13495.801044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13495.810678] LustreError: Skipped 1 previous similar message [13495.812686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13495.835872] Lustre: Skipped 1 previous similar message [13500.275698] Lustre: server umount lustre-OST0000 complete [13506.018691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13506.048248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13506.057715] Lustre: Skipped 2 previous similar messages [13511.137083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13511.151944] Lustre: Skipped 1 previous similar message [13515.235472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13515.251378] Lustre: Skipped 1 previous similar message [13519.329062] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13519.858787] Lustre: server umount lustre-MDT0000 complete [13525.489880] LustreError: 328765:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13525.515857] LustreError: 328765:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13525.764899] LustreError: 327314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789063639 with bad export cookie 10573823777125609928 [13525.781526] LustreError: 327314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13525.781923] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13526.591390] Lustre: server umount lustre-MDT0001 complete [13542.251387] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [13545.990625] Key type lgssc unregistered [13546.408850] LNet: 331204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13546.428742] LNetError: 331204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13546.454790] LNet: Removed LNI 192.168.204.150@tcp [13547.435470] Key type .llcrypt unregistered [13547.439066] Key type ._llcrypt unregistered [13570.620348] Key type ._llcrypt registered [13570.621978] Key type .llcrypt registered [13570.702277] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [13572.104959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13572.593535] alg: No test for adler32 (adler32-zlib) [13574.379249] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [13574.917411] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [13576.679203] Key type lgssc registered [13578.794037] Lustre: Echo OBD driver; http://www.lustre.org/ [13594.160066] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13594.191985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13595.912890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13602.077624] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13605.361822] LustreError: 332476:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13605.387370] LustreError: 332476:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13610.477392] LustreError: 332477:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13612.047610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13612.627743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13617.832738] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13626.553181] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13633.740936] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13642.975706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13643.525062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13650.969298] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13659.648361] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13676.196069] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13678.968031] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13685.729959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13685.738613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13685.750770] Lustre: Skipped 1 previous similar message [13685.768272] Lustre: Skipped 2 previous similar messages [13686.278503] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.50@tcp (stopping) [13690.861549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13690.864090] Lustre: Skipped 1 previous similar message [13691.335791] LustreError: 333445:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13693.239092] Lustre: server umount lustre-MDT0000 complete [13695.990632] LustreError: 332477:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13696.029082] LustreError: 332477:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13698.935086] LustreError: 332456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789063812 with bad export cookie 7368907426349456360 [13698.951713] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13698.952755] LustreError: 332456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13701.089543] LustreError: 332473:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13701.096607] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13701.112549] LustreError: 332473:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13701.124847] Lustre: Skipped 1 previous similar message [13701.127451] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13706.690348] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.50@tcp (stopping) [13706.703916] Lustre: Skipped 2 previous similar messages [13712.832746] LustreError: 332473:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13712.862497] LustreError: 332473:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [13713.375791] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13713.922855] Lustre: server umount lustre-MDT0001 complete [13720.381606] Lustre: server umount lustre-OST0000 complete [13731.767000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13738.663516] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13742.067264] Lustre: 336665:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.150@tcp [13745.171231] Lustre: server umount MGS complete [13758.366613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13759.417069] LustreError: 337237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13759.436714] LustreError: 337237:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13759.563829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13767.273216] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13780.467894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13780.924189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13787.252660] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13799.166996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13799.935369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13808.399356] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13818.344293] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13820.364408] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13828.581362] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13830.987382] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13840.868162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13840.878292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13840.886825] Lustre: Skipped 1 previous similar message [13840.887349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13840.912109] Lustre: Skipped 5 previous similar messages [13844.639414] Lustre: server umount lustre-OST0000 complete [13851.109684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13851.140513] Lustre: Skipped 1 previous similar message [13855.562878] Lustre: server umount lustre-MDT0000 complete [13856.226331] LustreError: 337238:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13856.238563] LustreError: 337238:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [13860.403935] LustreError: 337219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789063974 with bad export cookie 7368907426349457200 [13860.409037] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13860.413516] LustreError: 337219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [13860.858912] Lustre: server umount lustre-MDT0001 complete [13876.511428] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [13880.949884] Key type lgssc unregistered [13881.563520] LNet: 340675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13881.575981] LNetError: 340675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13881.605831] LNet: Removed LNI 192.168.204.150@tcp [13883.051396] Key type .llcrypt unregistered [13883.054658] Key type ._llcrypt unregistered [13905.159518] Key type ._llcrypt registered [13905.165927] Key type .llcrypt registered [13905.291417] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [13924.258839] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [13926.108332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13926.189503] alg: No test for adler32 (adler32-zlib) [13927.284682] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [13927.617845] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [13929.353100] Key type lgssc registered [13930.974440] Lustre: Echo OBD driver; http://www.lustre.org/ [13977.257141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13977.283604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13978.659792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13978.683915] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13978.776227] Lustre: lustre-MDT0000: new disk, initializing [13978.865274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13978.886105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13984.485750] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [13998.955893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13999.190245] Lustre: 343840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13999.304157] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13999.316483] Lustre: Skipped 1 previous similar message [13999.474597] Lustre: lustre-MDT0001: new disk, initializing [13999.648506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13999.729160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13999.747282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14006.000320] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [14019.424756] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14027.565734] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14038.471920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14038.931496] Lustre: lustre-OST0000: new disk, initializing [14038.945510] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14038.952805] Lustre: 345810:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14039.167993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14047.321570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14047.336342] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14047.485876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14048.369621] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [14062.574565] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14070.423639] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14070.948429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14075.912451] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14076.177248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14083.553505] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14083.575665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14083.585628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14083.593948] Lustre: Skipped 1 previous similar message [14083.618272] Lustre: Skipped 1 previous similar message [14088.689158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14088.705285] Lustre: Skipped 1 previous similar message [14093.279226] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14093.591342] Lustre: server umount lustre-OST0000 complete [14101.991953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14102.013991] LustreError: Skipped 1 previous similar message [14102.028510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14102.077233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14104.037875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14106.517031] Lustre: server umount lustre-MDT0000 complete [14114.390046] LustreError: 343830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064228 with bad export cookie 624153601705081048 [14114.398761] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14114.410453] LustreError: 343830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14115.066117] Lustre: server umount lustre-MDT0001 complete [14129.883881] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 14:17:21 (1789064241) [14156.384787] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [14174.357367] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [14225.544346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14226.098480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14226.173869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14226.392943] Lustre: lustre-MDT0000: new disk, initializing [14226.498057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14226.532207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14232.950945] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [14246.952276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14247.124502] Lustre: 350597:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14247.176886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14247.181756] Lustre: Skipped 1 previous similar message [14247.271619] Lustre: lustre-MDT0001: new disk, initializing [14247.407900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14247.469054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14247.482560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14254.667855] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [14266.393558] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14273.843986] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14283.464081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14284.130343] Lustre: lustre-OST0000: new disk, initializing [14284.139112] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14284.159712] Lustre: 352571:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14284.302384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14291.566721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14291.581905] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14291.796995] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14293.109162] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [14304.339699] Lustre: DEBUG MARKER: oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14348.264496] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14348.272430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14348.284332] LustreError: Skipped 1 previous similar message [14348.285837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14348.285846] Lustre: Skipped 2 previous similar messages [14348.327578] Lustre: Skipped 1 previous similar message [14353.257206] Lustre: server umount lustre-OST0000 complete [14358.502243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14358.513542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14358.520873] Lustre: Skipped 1 previous similar message [14358.531685] Lustre: Skipped 2 previous similar messages [14360.994338] Lustre: server umount lustre-MDT0000 complete [14363.622619] LustreError: 351619:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14363.652309] LustreError: 351619:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [14366.461779] LustreError: 350590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064480 with bad export cookie 624153601705081930 [14366.484692] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14367.012716] Lustre: server umount lustre-MDT0001 complete [14382.641466] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [14387.604678] Key type lgssc unregistered [14388.042720] LNet: 354747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14388.050679] LNetError: 354747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14388.072799] LNet: Removed LNI 192.168.204.150@tcp [14389.541204] Key type .llcrypt unregistered [14389.545094] Key type ._llcrypt unregistered [14506.810434] Key type ._llcrypt registered [14506.820169] Key type .llcrypt registered [14506.998514] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [14507.426327] Key type .llcrypt unregistered [14507.428085] Key type ._llcrypt unregistered [14602.080957] Key type ._llcrypt registered [14602.083812] Key type .llcrypt registered [14602.229893] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [14602.640466] Key type .llcrypt unregistered [14602.648895] Key type ._llcrypt unregistered [14638.732882] Key type ._llcrypt registered [14638.735423] Key type .llcrypt registered [14638.918464] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [14639.230600] Key type .llcrypt unregistered [14639.235218] Key type ._llcrypt unregistered [14679.796293] Key type ._llcrypt registered [14679.799579] Key type .llcrypt registered [14680.013677] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [14680.645227] Key type .llcrypt unregistered [14680.652932] Key type ._llcrypt unregistered [14779.871405] Key type ._llcrypt registered [14779.874256] Key type .llcrypt registered [14779.957263] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [14780.342814] Key type .llcrypt unregistered [14780.348186] Key type ._llcrypt unregistered [14816.668506] Key type ._llcrypt registered [14816.676910] Key type .llcrypt registered [14816.814502] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [14817.150359] Key type .llcrypt unregistered [14817.166899] Key type ._llcrypt unregistered [14851.340521] Key type ._llcrypt registered [14851.342728] Key type .llcrypt registered [14851.474799] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [14851.804507] Key type .llcrypt unregistered [14851.806465] Key type ._llcrypt unregistered [14884.398807] Key type ._llcrypt registered [14884.401337] Key type .llcrypt registered [14884.536856] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [14885.026295] Key type .llcrypt unregistered [14885.027596] Key type ._llcrypt unregistered [14921.058112] Key type ._llcrypt registered [14921.067634] Key type .llcrypt registered [14921.270831] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [14921.645692] Key type .llcrypt unregistered [14921.658698] Key type ._llcrypt unregistered [14960.140948] Key type ._llcrypt registered [14960.147495] Key type .llcrypt registered [14960.385965] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [14961.077777] Key type .llcrypt unregistered [14961.079505] Key type ._llcrypt unregistered [15038.824914] Key type ._llcrypt registered [15038.829839] Key type .llcrypt registered [15039.012771] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [15039.391466] Key type .llcrypt unregistered [15039.396640] Key type ._llcrypt unregistered [15074.378617] Key type ._llcrypt registered [15074.387324] Key type .llcrypt registered [15074.665622] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [15075.129283] Key type .llcrypt unregistered [15075.132979] Key type ._llcrypt unregistered [15112.420166] Key type ._llcrypt registered [15112.428828] Key type .llcrypt registered [15112.596727] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [15113.020418] Key type .llcrypt unregistered [15113.024452] Key type ._llcrypt unregistered [15178.175866] Key type ._llcrypt registered [15178.182975] Key type .llcrypt registered [15178.327849] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [15196.236783] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [15197.837264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15197.934074] alg: No test for adler32 (adler32-zlib) [15199.181376] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [15199.584458] LNet: Added LNI 192.168.204.150@tcp [8/256/0/180] [15201.415219] Key type lgssc registered [15202.956886] Lustre: Echo OBD driver; http://www.lustre.org/ [15248.000703] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15248.036518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15249.576210] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [15249.626245] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [15249.804368] Lustre: tst250fs-MDT0000: new disk, initializing [15249.910825] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15249.941528] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15256.684836] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [15271.672500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15271.953673] Lustre: 375079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15272.128626] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [15272.141744] Lustre: Skipped 1 previous similar message [15272.279670] Lustre: tst250fs-MDT0001: new disk, initializing [15272.523905] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15272.663043] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15272.699481] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15279.991777] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [15286.841484] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [15295.307415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15295.660193] Lustre: tst250fs-OST0000: new disk, initializing [15295.665533] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [15295.672224] Lustre: 376717:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15295.816858] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15303.504712] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [15303.823397] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15303.843460] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15303.999705] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15314.748697] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15314.960910] Lustre: tst250fs-OST0001: new disk, initializing [15314.964607] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [15314.970110] Lustre: 377587:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15315.068739] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [15321.942581] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_default_debug -1 all [15324.793516] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [15324.809186] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [15324.948391] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [15329.212332] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [15331.315740] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [15331.321771] Lustre: Skipped 1 previous similar message [15355.881083] Lustre: tst250fs-MDT0000-lwp-OST0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15355.888195] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [15355.900256] Lustre: Skipped 2 previous similar messages [15355.915954] Lustre: Skipped 3 previous similar messages [15360.982588] Lustre: server umount tst250fs-MDT0000 complete [15365.545966] LustreError: 375072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065479 with bad export cookie 547792469150460872 [15365.549306] LustreError: MGC192.168.204.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15365.564744] LustreError: 375072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15366.056636] Lustre: server umount tst250fs-MDT0001 complete [15381.433052] Lustre: server umount tst250fs-OST0000 complete [15385.571330] Lustre: 373299:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789065483/real 1789065483] req@ffff9df2b4df8e00 x1875970949719168/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789065499 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15385.646428] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15385.673034] Lustre: Skipped 1 previous similar message [15386.593543] Lustre: 373301:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789065484/real 1789065484] req@ffff9df28748f100 x1875970949719424/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789065500 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15387.088673] Lustre: server umount tst250fs-OST0001 complete [15415.209367] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [15428.430434] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [15474.142641] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 14:39:46 (1789065586) [15476.431173] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [15500.163642] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing set_hostid [15511.363741] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing load_modules_local [15553.513897] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15279 sec ========= 14:41:05 (1789065665) [15556.157497] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:41:07 (1789065667) === [15561.029929] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:41:13 (1789065673) === [15611.407929] Lustre: DEBUG MARKER: oleg450-server.virtnet: executing unload_modules_local [15614.738305] Key type lgssc unregistered [15615.095197] LNet: 386739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15615.109204] LNetError: 386739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15615.141925] LNet: Removed LNI 192.168.204.150@tcp [15616.298284] Key type .llcrypt unregistered [15616.301867] Key type ._llcrypt unregistered