[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 461249870 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002414] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007582] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010136] LSM: Security Framework initializing [ 0.011063] Yama: becoming mindful. [ 0.012036] SELinux: Initializing. [ 0.013059] *** VALIDATE selinux *** [ 0.021035] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025815] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027119] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029007] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030123] *** VALIDATE tmpfs *** [ 0.031409] *** VALIDATE proc *** [ 0.032235] *** VALIDATE cgroup *** [ 0.033012] *** VALIDATE cgroup2 *** [ 0.035077] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038033] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042043] debug: unmapping init [mem 0xffffffffb2459000-0xffffffffb2460fff] [ 0.044875] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045618] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046020] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048015] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053250] rcu: Hierarchical SRCU implementation. [ 0.055322] smp: Bringing up secondary CPUs ... [ 0.056588] x86: Booting SMP configuration: [ 0.057026] .... node #0, CPUs: #1 #2 #3 [ 0.060305] smp: Brought up 1 node, 4 CPUs [ 0.062013] smpboot: Max logical packages: 1 [ 0.063019] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.099930] node 0 deferred pages initialised in 35ms [ 0.104010] devtmpfs: initialized [ 0.105386] x86/mm: Memory block size: 128MB [ 0.107914] gcov: version magic: 0x41383552 [ 0.109182] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.110083] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.111285] pinctrl core: initialized pinctrl subsystem [ 0.112303] [ 0.113025] ************************************************************* [ 0.114017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.115016] ** ** [ 0.116018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.117020] ** ** [ 0.118021] ** This means that this kernel is built to expose internal ** [ 0.119027] ** IOMMU data structures, which may compromise security on ** [ 0.120019] ** your system. ** [ 0.121018] ** ** [ 0.122016] ** If you see this message and you are not debugging the ** [ 0.123015] ** kernel, report this immediately to your vendor! ** [ 0.124018] ** ** [ 0.125018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.126016] ************************************************************* [ 0.127823] NET: Registered protocol family 16 [ 0.128522] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.129064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.130064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.131552] cpuidle: using governor menu [ 0.132000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.137610] PCI: Using configuration type 1 for base access [ 0.140120] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.150168] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.153064] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.157075] cryptd: max_cpu_qlen set to 1000 [ 0.159236] ACPI: Added _OSI(Module Device) [ 0.160018] ACPI: Added _OSI(Processor Device) [ 0.161016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.162016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.166214] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.168568] ACPI: Interpreter enabled [ 0.169081] ACPI: PM: (supports S0 S3 S4 S5) [ 0.170041] ACPI: Using IOAPIC for interrupt routing [ 0.171175] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.175546] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.185859] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.188036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.190014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.192086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.198368] acpiphp: Slot [2] registered [ 0.200103] acpiphp: Slot [5] registered [ 0.201128] acpiphp: Slot [6] registered [ 0.203128] acpiphp: Slot [7] registered [ 0.204198] acpiphp: Slot [8] registered [ 0.206118] acpiphp: Slot [9] registered [ 0.208119] acpiphp: Slot [10] registered [ 0.209170] acpiphp: Slot [3] registered [ 0.211110] acpiphp: Slot [4] registered [ 0.213105] acpiphp: Slot [11] registered [ 0.214119] acpiphp: Slot [12] registered [ 0.216096] acpiphp: Slot [13] registered [ 0.217110] acpiphp: Slot [14] registered [ 0.219097] acpiphp: Slot [15] registered [ 0.220105] acpiphp: Slot [16] registered [ 0.221084] acpiphp: Slot [17] registered [ 0.223101] acpiphp: Slot [18] registered [ 0.224119] acpiphp: Slot [19] registered [ 0.226082] acpiphp: Slot [20] registered [ 0.227079] acpiphp: Slot [21] registered [ 0.228075] acpiphp: Slot [22] registered [ 0.229109] acpiphp: Slot [23] registered [ 0.230132] acpiphp: Slot [24] registered [ 0.231041] acpiphp: Slot [25] registered [ 0.233116] acpiphp: Slot [26] registered [ 0.235104] acpiphp: Slot [27] registered [ 0.237103] acpiphp: Slot [28] registered [ 0.238084] acpiphp: Slot [29] registered [ 0.240169] acpiphp: Slot [30] registered [ 0.242115] acpiphp: Slot [31] registered [ 0.243060] PCI host bridge to bus 0000:00 [ 0.245021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.247020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.250020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.253025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.256024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.259027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.261179] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.264091] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.267427] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.277908] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.285060] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.288018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.290018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.293019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.296221] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.298982] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.302043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.304974] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.310015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.323015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.329016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.334875] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.340013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.347019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.362016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.370701] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.377014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.387016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.404015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.414220] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.420013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.425014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.439014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.449000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.454016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.461016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.476061] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.488160] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.500019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.508071] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.523018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.534372] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.542017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.549016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.566017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.575973] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.577212] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.579262] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.581377] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.584217] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.590046] iommu: Default domain type: Passthrough [ 0.591458] SCSI subsystem initialized [ 0.593126] ACPI: bus type USB registered [ 0.595114] usbcore: registered new interface driver usbfs [ 0.597080] usbcore: registered new interface driver hub [ 0.598082] usbcore: registered new device driver usb [ 0.600182] pps_core: LinuxPPS API ver. 1 registered [ 0.602012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.605081] PTP clock support registered [ 0.607120] EDAC MC: Ver: 3.0.0 [ 0.609126] PCI: Using ACPI for IRQ routing [ 0.610764] NetLabel: Initializing [ 0.612013] NetLabel: domain hash size = 128 [ 0.613020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.614086] NetLabel: unlabeled traffic allowed by default [ 0.616267] vgaarb: loaded [ 0.618044] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.619010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.623158] clocksource: Switched to clocksource kvm-clock [ 0.726629] VFS: Disk quotas dquot_6.6.0 [ 0.728870] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.731715] *** VALIDATE ramfs *** [ 0.733205] *** VALIDATE hugetlbfs *** [ 0.734920] pnp: PnP ACPI init [ 0.737579] pnp: PnP ACPI: found 6 devices [ 0.758407] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.762238] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.764861] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.767036] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.769603] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.772052] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.774818] NET: Registered protocol family 2 [ 0.777574] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.783248] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.787350] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.793204] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.796975] TCP: Hash tables configured (established 65536 bind 65536) [ 0.800422] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.803847] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.807240] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.810154] NET: Registered protocol family 1 [ 0.813547] RPC: Registered named UNIX socket transport module. [ 0.816308] RPC: Registered udp transport module. [ 0.818034] RPC: Registered tcp transport module. [ 0.820423] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.823111] NET: Registered protocol family 44 [ 0.824702] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.826858] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.829012] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.831476] PCI: CLS 0 bytes, default 64 [ 0.832990] Unpacking initramfs... [ 2.291611] debug: unmapping init [mem 0xffff8bc63cc54000-0xffff8bc63ffbffff] [ 2.296057] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.298841] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.302306] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.797646] Initialise system trusted keyrings [ 2.799254] Key type blacklist registered [ 2.801335] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.818915] zbud: loaded [ 2.830166] *** VALIDATE nfs *** [ 2.832363] *** VALIDATE nfs4 *** [ 2.835723] pstore: using deflate compression [ 2.841905] Platform Keyring initialized [ 2.947530] NET: Registered protocol family 38 [ 2.949773] Key type asymmetric registered [ 2.951760] Asymmetric key parser 'x509' registered [ 2.953767] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.956707] io scheduler mq-deadline registered [ 2.958781] io scheduler kyber registered [ 2.960747] io scheduler bfq registered [ 2.962770] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.965804] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.968552] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.971834] ACPI: Power Button [PWRF] [ 2.979182] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.986612] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.998809] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.005674] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.022165] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.050237] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.079031] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.083908] Non-volatile memory driver v1.3 [ 3.085984] Linux agpgart interface v0.103 [ 3.120805] virtio_blk virtio1: [vda] 150400 512-byte logical blocks (77.0 MB/73.4 MiB) [ 3.125623] vda: detected capacity change from 0 to 77004800 [ 3.143244] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.146613] vdb: detected capacity change from 0 to 1073741824 [ 3.160611] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.164958] vdc: detected capacity change from 0 to 2621440000 [ 3.180249] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.184035] vdd: detected capacity change from 0 to 2621440000 [ 3.200640] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.203959] vde: detected capacity change from 0 to 4294967296 [ 3.218362] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.221462] vdf: detected capacity change from 0 to 4294967296 [ 3.227410] libphy: Fixed MDIO Bus: probed [ 3.237307] usbcore: registered new interface driver usbserial_generic [ 3.240636] usbserial: USB Serial support registered for generic [ 3.245182] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.250588] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.252765] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.255483] mousedev: PS/2 mouse device common for all mice [ 3.259563] rtc_cmos 00:05: RTC can wake from S4 [ 3.263591] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.263770] rtc_cmos 00:05: registered as rtc0 [ 3.270203] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.271541] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.273617] intel_pstate: CPU model not supported [ 3.278420] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.282289] hid: raw HID events driver (C) Jiri Kosina [ 3.284669] usbcore: registered new interface driver usbhid [ 3.286649] usbhid: USB HID core driver [ 3.288328] drop_monitor: Initializing network drop monitor service [ 3.290623] Initializing XFRM netlink socket [ 3.292581] NET: Registered protocol family 10 [ 3.295903] Segment Routing with IPv6 [ 3.297493] NET: Registered protocol family 17 [ 3.299352] mpls_gso: MPLS GSO support [ 3.305754] RAS: Correctable Errors collector initialized. [ 3.308336] AVX version of gcm_enc/dec engaged. [ 3.310206] AES CTR mode by8 optimization enabled [ 3.396583] sched_clock: Marking stable (3396545555, 0)->(4374793531, -978247976) [ 3.400792] registered taskstats version 1 [ 3.402654] Loading compiled-in X.509 certificates [ 3.404920] zswap: loaded using pool lzo/zbud [ 3.433373] Key type big_key registered [ 3.446281] Key type encrypted registered [ 3.448150] ima: No TPM chip found, activating TPM-bypass! [ 3.450957] ima: Allocated hash algorithm: sha1 [ 3.453158] ima: No architecture policies found [ 3.455351] evm: Initialising EVM extended attributes: [ 3.457688] evm: security.selinux [ 3.459190] evm: security.ima [ 3.460528] evm: security.capability [ 3.462019] evm: HMAC attrs: 0x1 [ 3.465853] rtc_cmos 00:05: setting system clock to 2026-10-02 01:57:14 UTC (1790906234) [ 3.472711] debug: unmapping init [mem 0xffffffffb3403000-0xffffffffb35fffff] [ 3.476193] debug: unmapping init [mem 0xffffffffb2182000-0xffffffffb2458fff] [ 3.489139] Write protecting the kernel read-only data: 28672k [ 3.493176] debug: unmapping init [mem 0xffffffffb0803000-0xffffffffb09fffff] [ 3.495483] debug: unmapping init [mem 0xffffffffb1114000-0xffffffffb11fffff] [ 3.528820] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.536278] systemd[1]: Detected virtualization kvm. [ 3.538094] systemd[1]: Detected architecture x86-64. [ 3.539753] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.566514] systemd[1]: No hostname configured. [ 3.568466] systemd[1]: Set hostname to . [ 3.570630] random: systemd: uninitialized urandom read (16 bytes read) [ 3.573386] systemd[1]: Initializing machine ID from random generator. [ 3.617280] random: ln: uninitialized urandom read (6 bytes read) [ 3.701648] random: systemd: uninitialized urandom read (16 bytes read) [ 3.704714] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.709241] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.714344] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.316637] device-mapper: uevent: version 1.0.3 [ 4.318816] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.045916] virtio_net virtio0 ens2: renamed from eth0 [ 5.046810] random: fast init done [ 5.062374] scsi host0: ata_piix [ 5.124625] scsi host1: ata_piix [ 5.126772] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.130237] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.814366] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.967803] random: crng init done [ 9.969443] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.326939] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.423319] printk: systemd: 24 output lines suppressed due to ratelimiting [ 11.674259] SELinux: Disabled at runtime. [ 11.735290] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.744065] systemd[1]: Detected virtualization kvm. [ 11.747456] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.249182] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.253628] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.259068] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.263384] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.268371] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.278337] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.285584] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ 12.355650] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.740121] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.000413] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.040158] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.236785] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.247710] EDAC sbridge: Ver: 1.1.2 [ 15.049489] Key type dns_resolver registered [ 15.369846] NFS: Registering the id_resolver key type [ 15.372272] Key type id_resolver registered [ 15.374204] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg353-server login: [ 41.802847] libcfs: loading out-of-tree module taints kernel. [ 41.822601] Key type ._llcrypt registered [ 41.824289] Key type .llcrypt registered [ 41.882858] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 54.696375] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 56.450797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 56.466238] alg: No test for adler32 (adler32-zlib) [ 58.003503] Lustre: Lustre: Build Version: 2.17.58_63_g006d100 [ 59.024567] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 60.943911] Key type lgssc registered [ 63.033320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 77.013010] hrtimer: interrupt took 3271514 ns [ 80.034499] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 127.472927] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 145.334782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 145.378124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 146.882528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 146.938840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 147.078000] Lustre: lustre-MDT0000: new disk, initializing [ 147.211364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 147.247354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 152.143832] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.826443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.015488] Lustre: 6489:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 169.068571] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 169.073906] Lustre: Skipped 1 previous similar message [ 169.189512] Lustre: lustre-MDT0001: new disk, initializing [ 169.298601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 169.347505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 169.361926] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 174.031893] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.463737] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 187.484874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 187.764655] Lustre: lustre-OST0000: new disk, initializing [ 187.769337] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 187.781871] Lustre: 8425:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 187.878783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.132185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 194.157661] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 194.319826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 194.898377] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.215757] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 212.516537] Lustre: lustre-OST0001: new disk, initializing [ 212.537190] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 212.552342] Lustre: 9498:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 212.687600] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 220.228022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 220.238562] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 220.329677] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 220.461516] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.371870] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 243.438860] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 251.187881] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing check_logdir /tmp/testlogs/ [ 256.584598] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing yml_node [ 261.672549] Lustre: DEBUG MARKER: Client: 2.17.58.63 [ 265.006544] Lustre: DEBUG MARKER: MDS: 2.17.58.63 [ 268.345430] Lustre: DEBUG MARKER: OSS: 2.17.58.63 [ 270.769397] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Oct 1 22:01:40 EDT 2026 [ 290.699322] Lustre: DEBUG MARKER: only running tests: 121-999 [ 292.197736] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 293.779963] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 312.802490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.817032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 312.817154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 312.818367] Lustre: Skipped 2 previous similar messages [ 312.859342] Lustre: Skipped 3 previous similar messages [ 316.359292] Lustre: server umount lustre-MDT0000 complete [ 319.721750] LustreError: 6480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790906550 with bad export cookie 18252827007869943440 [ 319.728207] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 319.732140] LustreError: 6480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 323.040840] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.042495] LustreError: 6497:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 323.046690] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 323.063324] Lustre: Skipped 2 previous similar messages [ 323.087496] LustreError: 6497:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 324.064629] LustreError: 6495:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.064901] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 324.083883] LustreError: 6495:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 324.090885] Lustre: Skipped 2 previous similar messages [ 329.186205] LustreError: 6496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 329.189340] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 329.217569] LustreError: 6496:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 329.231240] Lustre: Skipped 1 previous similar message [ 334.303162] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 334.304995] LustreError: 7913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 334.309558] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 334.344705] LustreError: 7913:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 334.716531] Lustre: server umount lustre-MDT0001 complete [ 338.708262] Lustre: server umount lustre-OST0000 complete [ 342.994166] Lustre: server umount lustre-OST0001 complete [ 351.442212] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 364.252269] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 405.469435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 405.759646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 405.786543] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 405.890172] Lustre: lustre-MDT0000: new disk, initializing [ 405.983222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 406.003220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 411.664475] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 423.086651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 423.275667] Lustre: 15119:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 423.369668] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 423.375205] Lustre: Skipped 1 previous similar message [ 423.508896] Lustre: lustre-MDT0001: new disk, initializing [ 423.670481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 423.731084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 423.748945] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 429.121990] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 438.639991] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 445.735510] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 453.862673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 454.343326] Lustre: lustre-OST0000: new disk, initializing [ 454.346723] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 454.353403] Lustre: 17087:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 454.478388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 456.065249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 456.089118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 456.318868] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 462.012037] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 473.366953] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 478.925964] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 479.213401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 484.477426] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 484.716777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 486.880725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 486.884553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 486.897548] LustreError: Skipped 1 previous similar message [ 486.902943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 486.915220] Lustre: Skipped 1 previous similar message [ 486.943204] Lustre: Skipped 2 previous similar messages [ 492.997421] Lustre: server umount lustre-OST0000 complete [ 497.120788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 503.196977] Lustre: server umount lustre-MDT0000 complete [ 507.313366] LustreError: 15113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790906738 with bad export cookie 18252827007869944973 [ 507.331798] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 507.335431] LustreError: 15113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 507.914090] Lustre: server umount lustre-MDT0001 complete [ 514.206653] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 22:05:44 (1790906744) [ 534.964967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 535.630051] LustreError: 19964:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 535.701947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 540.743372] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 545.254558] LustreError: 19965:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.277487] LustreError: 19965:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 550.417179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.210515] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 566.371894] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 572.359536] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 574.643719] Lustre: Failing over lustre-MDT0000 [ 575.105930] Lustre: server umount lustre-MDT0000 complete [ 575.458283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 575.481970] LustreError: 21204:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 575.486657] Lustre: Skipped 2 previous similar messages [ 575.515301] LustreError: 21204:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 591.839128] Lustre: 3622:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790906806/real 1790906806] req@ffff8bc69128b100 x1877901354105600/t0(0) o400->MGC192.168.203.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790906822 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 591.868602] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 595.526823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 602.082692] LustreError: 3619:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8bc5876c9500 x1877901354108672/t0(0) o250->MGC192.168.203.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 602.491655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 602.502684] Lustre: Skipped 1 previous similar message [ 602.567274] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 607.713290] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 607.735833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 607.738690] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 608.312562] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 621.042496] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 626.147149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 626.162248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 626.165030] Lustre: Skipped 7 previous similar messages [ 631.654840] Lustre: server umount lustre-MDT0000 complete [ 636.387638] LustreError: 19960:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 636.408135] LustreError: 19960:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 636.525659] LustreError: 21533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790906867 with bad export cookie 18252827007869946387 [ 636.533936] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 636.542536] LustreError: 21533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 636.957818] Lustre: server umount lustre-MDT0001 complete [ 647.260771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 653.162569] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 662.888882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 668.523226] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 675.791417] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 682.302947] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 685.554719] Lustre: Failing over lustre-MDT0000 [ 686.143553] Lustre: server umount lustre-MDT0000 complete [ 688.673254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 688.685560] Lustre: Skipped 2 previous similar messages [ 703.975090] LustreError: 23305:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 704.025946] LustreError: 23305:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 704.868484] Lustre: 3620:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790906919/real 1790906919] req@ffff8bc6c1af6a00 x1877901354135936/t0(0) o400->MGC192.168.203.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790906935 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 704.915349] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 706.762695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 714.214071] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfd4f1174bebe8dbd [ 714.240475] Lustre: MGC192.168.203.153@tcp: Connection restored to 0@lo (at 0@lo) [ 714.246154] Lustre: Skipped 1 previous similar message [ 714.640383] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 719.527738] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 719.844024] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 719.863167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 719.891281] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 730.560924] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 735.201535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 735.216146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 735.240250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 735.245373] Lustre: Skipped 2 previous similar messages [ 741.101032] Lustre: server umount lustre-MDT0000 complete [ 744.946243] LustreError: 23284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790906975 with bad export cookie 18252827007869947325 [ 744.965814] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 744.966699] LustreError: 23284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 745.364808] Lustre: server umount lustre-MDT0001 complete [ 754.387506] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 22:09:44 (1790906984) [ 770.837043] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 782.150696] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 825.169839] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 837.412233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 837.775983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 837.828743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 837.902629] Lustre: lustre-MDT0000: new disk, initializing [ 837.967136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 837.977789] Lustre: Skipped 3 previous similar messages [ 837.997710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 843.085373] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 854.139932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 854.281636] Lustre: 29678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 854.618118] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 859.370201] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 869.596747] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 876.030581] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 883.606054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 883.971120] Lustre: 31645:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 886.015545] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 886.063451] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 886.168458] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 886.180050] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 890.366219] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 900.416648] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 906.297495] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 927.202914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 927.206684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 927.214760] Lustre: Skipped 1 previous similar message [ 927.221735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 927.221741] Lustre: Skipped 2 previous similar messages [ 933.175688] Lustre: server umount lustre-OST0000 complete [ 942.870918] Lustre: server umount lustre-MDT0000 complete [ 946.646067] LustreError: 29670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790907177 with bad export cookie 18252827007869947738 [ 946.659184] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 946.669872] LustreError: 29670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 946.989446] Lustre: server umount lustre-MDT0001 complete [ 956.815789] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 959.669488] Key type lgssc unregistered [ 960.132223] LNet: 33625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 960.145618] LNetError: 33625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 960.164764] LNet: Removed LNI 192.168.203.153@tcp [ 961.009393] Key type .llcrypt unregistered [ 961.011657] Key type ._llcrypt unregistered [ 983.072307] Key type ._llcrypt registered [ 983.074974] Key type .llcrypt registered [ 983.191041] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 983.937615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 984.008446] alg: No test for adler32 (adler32-zlib) [ 985.127925] Lustre: Lustre: Build Version: 2.17.58_63_g006d100 [ 985.434307] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 987.063217] Key type lgssc registered [ 988.378515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 996.470772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 996.495973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 998.239381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1004.506291] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1008.611240] LustreError: 35053:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1008.630791] LustreError: 35053:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1013.736112] LustreError: 35054:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1014.892065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1015.401623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1021.061373] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1028.721427] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1034.259701] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1041.508911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1042.060121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1048.902831] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1057.253914] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1060.461193] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1077.683967] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 22:15:07 (1790907307) [ 1079.571507] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1081.445308] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1088.490386] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 22:15:18 (1790907318) [ 1099.052620] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 22:15:28 (1790907328) [ 1100.021497] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1108.398607] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1108.404520] Lustre: Skipped 2 previous similar messages [ 1110.519288] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 22:15:40 (1790907340) [ 1112.756953] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1112.764542] Lustre: Skipped 2 previous similar messages [ 1121.243626] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1121.259627] Lustre: Skipped 4 previous similar messages [ 1139.477791] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 22:16:09 (1790907369) [ 1142.311727] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1146.353901] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1155.287400] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 22:16:25 (1790907385) [ 1157.400864] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1157.410576] Lustre: Skipped 6 previous similar messages [ 1170.668720] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 22:16:40 (1790907400) [ 1187.810180] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1187.816162] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1187.823178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1188.836822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1188.839106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1188.847169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.438060] Lustre: server umount lustre-OST0000 complete [ 1199.077505] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1199.086712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1199.102176] Lustre: Skipped 1 previous similar message [ 1199.112855] Lustre: Skipped 2 previous similar messages [ 1204.194389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1204.200852] Lustre: Skipped 1 previous similar message [ 1208.294844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1208.301605] Lustre: Skipped 1 previous similar message [ 1210.336128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1210.548021] Lustre: server umount lustre-MDT0000 complete [ 1213.410510] LustreError: 35054:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1213.440617] LustreError: 35054:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1214.203826] LustreError: 35035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790907445 with bad export cookie 15492838258725666653 [ 1214.206384] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1214.211548] LustreError: 35035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1214.541918] Lustre: server umount lustre-MDT0001 complete [ 1224.727531] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 1227.579066] Key type lgssc unregistered [ 1227.900332] LNet: 42919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1227.908941] LNetError: 42919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1227.935412] LNet: Removed LNI 192.168.203.153@tcp [ 1228.703384] Key type .llcrypt unregistered [ 1228.705911] Key type ._llcrypt unregistered [ 1245.496345] Key type ._llcrypt registered [ 1245.500290] Key type .llcrypt registered [ 1245.627179] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 1261.532550] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 1262.542157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1262.577529] alg: No test for adler32 (adler32-zlib) [ 1263.886505] Lustre: Lustre: Build Version: 2.17.58_63_g006d100 [ 1264.233379] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 1265.975261] Key type lgssc registered [ 1267.346257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1305.096275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1305.148289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1306.397330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1306.416160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1306.477934] Lustre: lustre-MDT0000: new disk, initializing [ 1306.536597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1306.562246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1311.759530] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1324.286845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1324.464737] Lustre: 46086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1324.566935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1324.571420] Lustre: Skipped 1 previous similar message [ 1324.619282] Lustre: lustre-MDT0001: new disk, initializing [ 1324.717585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1324.765620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1324.783341] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1329.774255] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1338.658090] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1343.984894] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1350.522546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1350.829869] Lustre: lustre-OST0000: new disk, initializing [ 1350.833864] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1350.841683] Lustre: 48060:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1350.928034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1356.336599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1356.347397] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1356.457791] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1356.686557] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1365.665118] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1370.173950] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1370.438230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1375.462568] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1375.703205] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1381.856541] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1381.867843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1381.913379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1382.370392] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1382.380751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1383.862762] Lustre: server umount lustre-OST0000 complete [ 1391.585058] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1391.593544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1391.613433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1391.623497] Lustre: Skipped 2 previous similar messages [ 1394.646981] Lustre: server umount lustre-MDT0000 complete [ 1398.173706] LustreError: 46886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790907629 with bad export cookie 5913521942107115308 [ 1398.179625] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1398.198392] LustreError: 46886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1398.634088] Lustre: server umount lustre-MDT0001 complete [ 1409.663289] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 22:20:39 (1790907639) [ 1416.787292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1417.410117] LustreError: 50518:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1417.434458] LustreError: 50518:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1417.478880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.972794] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1422.827040] LustreError: 50519:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1427.939563] LustreError: 50518:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1430.005620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1430.367856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1435.269841] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1442.988657] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1448.604864] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1455.490674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1455.971370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1461.751291] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1469.613918] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1477.648881] LustreError: 53048:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1478.500578] Lustre: Modifying parameter general.timeout=1 in log params [ 1479.972879] Lustre: Modifying parameter general.timeout=3 in log params [ 1479.975515] Lustre: Skipped 1 previous similar message [ 1482.220770] Lustre: Modifying parameter general.timeout=6 in log params [ 1482.224960] Lustre: Skipped 2 previous similar messages [ 1486.492581] Lustre: Modifying parameter general.timeout=11 in log params [ 1486.499289] Lustre: Skipped 4 previous similar messages [ 1494.980961] Lustre: Modifying parameter general.timeout=21 in log params [ 1494.986925] Lustre: Skipped 9 previous similar messages [ 1511.257813] Lustre: Modifying parameter general.timeout=38 in log params [ 1511.272411] Lustre: Skipped 16 previous similar messages [ 1543.411469] Lustre: Modifying parameter general.timeout=73 in log params [ 1543.415309] Lustre: Skipped 34 previous similar messages [ 1607.105992] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 22:23:57 (1790907837) [ 1607.990249] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1607.996292] Lustre: Skipped 57 previous similar messages [ 1609.802851] Lustre: 59996:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1611.750284] Lustre: 60094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1615.469780] Lustre: 60292:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1615.483188] Lustre: 60292:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1619.992311] Lustre: 60505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1620.007991] Lustre: 60505:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1648.302724] Lustre: 61859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1648.322235] LustreError: 61859:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1652.595564] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 22:24:42 (1790907882) [ 1665.506678] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1665.516243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1665.526175] Lustre: Skipped 2 previous similar messages [ 1665.533790] Lustre: Skipped 2 previous similar messages [ 1670.189929] Lustre: server umount lustre-MDT0000 complete [ 1670.631569] LustreError: 50519:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1670.648695] LustreError: 50519:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1673.922526] LustreError: 53691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790907904 with bad export cookie 5913521942107116190 [ 1673.924181] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1673.939431] LustreError: 53691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1674.276101] Lustre: server umount lustre-MDT0001 complete [ 1688.638486] Lustre: server umount lustre-OST0000 complete [ 1716.459643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1716.795864] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1717.117484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1722.502491] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1732.669117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1732.823407] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1732.878548] Lustre: 64392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1737.721164] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1744.565439] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1750.623333] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1757.316941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1757.585148] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1757.930818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1757.940540] Lustre: Skipped 1 previous similar message [ 1765.610136] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1776.728493] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1778.938863] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1778.949197] Lustre: Skipped 22 previous similar messages [ 1779.029848] Lustre: 66956:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1788.388249] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1788.390858] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1788.396530] LustreError: Skipped 1 previous similar message [ 1788.397076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1788.397079] Lustre: Skipped 2 previous similar messages [ 1788.424662] Lustre: Skipped 2 previous similar messages [ 1793.506153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1793.515856] Lustre: Skipped 2 previous similar messages [ 1793.963847] Lustre: server umount lustre-OST0000 complete [ 1798.625070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1798.654833] Lustre: Skipped 1 previous similar message [ 1803.746478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1803.765857] Lustre: Skipped 3 previous similar messages [ 1804.742472] Lustre: server umount lustre-MDT0000 complete [ 1808.473702] LustreError: 64384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790908039 with bad export cookie 5913521942107119207 [ 1808.483695] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1808.487046] LustreError: 64384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1809.161379] Lustre: server umount lustre-MDT0001 complete [ 1821.078326] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 1824.029057] Key type lgssc unregistered [ 1824.295927] LNet: 68163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1824.308955] LNetError: 68163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1824.333599] LNet: Removed LNI 192.168.203.153@tcp [ 1825.232994] Key type .llcrypt unregistered [ 1825.240218] Key type ._llcrypt unregistered [ 1849.549968] Key type ._llcrypt registered [ 1849.552101] Key type .llcrypt registered [ 1849.658502] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 1850.706466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1850.760444] alg: No test for adler32 (adler32-zlib) [ 1851.765121] Lustre: Lustre: Build Version: 2.17.58_63_g006d100 [ 1852.029851] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 1853.687191] Key type lgssc registered [ 1854.967310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1862.693601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1862.711354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1864.243174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1869.336959] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1874.404173] LustreError: 69597:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1874.417964] LustreError: 69597:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1877.508138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1878.023365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1882.175258] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1890.102995] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1895.992551] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1903.168870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1903.802404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1909.529595] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1916.531653] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1928.112521] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1937.890596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1937.905135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1937.915680] Lustre: Skipped 2 previous similar messages [ 1942.722083] Lustre: server umount lustre-MDT0000 complete [ 1943.013162] LustreError: 69596:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1943.032854] LustreError: 69596:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1945.982485] LustreError: 70337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790908177 with bad export cookie 821208315646676974 [ 1945.985830] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1945.992461] LustreError: 70337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1946.195912] Lustre: server umount lustre-MDT0001 complete [ 1960.537986] Lustre: server umount lustre-OST0000 complete [ 1987.187451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1987.483581] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1987.506536] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1987.742914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1992.028476] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1999.864189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1999.931141] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1999.950293] Lustre: 74465:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2000.404597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2003.639492] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2010.311531] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2016.399991] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2023.418493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2023.600779] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2024.077250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2030.382514] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2036.870496] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2038.423311] Lustre: 77043:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2040.068720] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2044.898179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2044.907653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2044.933091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2044.941017] Lustre: Skipped 2 previous similar messages [ 2045.414788] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2050.194735] Lustre: server umount lustre-OST0000 complete [ 2055.649666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.662136] Lustre: Skipped 1 previous similar message [ 2055.665693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2055.669511] Lustre: Skipped 3 previous similar messages [ 2059.604123] Lustre: server umount lustre-MDT0000 complete [ 2060.769089] LustreError: 74478:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2060.793381] LustreError: 74478:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2062.837643] LustreError: 75212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790908293 with bad export cookie 821208315646677807 [ 2062.846227] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2062.847740] LustreError: 75212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2063.134985] Lustre: server umount lustre-MDT0001 complete [ 2072.766356] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 2075.486835] Key type lgssc unregistered [ 2075.755508] LNet: 78249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2075.764689] LNetError: 78249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2075.782845] LNet: Removed LNI 192.168.203.153@tcp [ 2076.535826] Key type .llcrypt unregistered [ 2076.540288] Key type ._llcrypt unregistered [ 2098.622752] Key type ._llcrypt registered [ 2098.629463] Key type .llcrypt registered [ 2098.720547] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2099.728085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2099.751198] alg: No test for adler32 (adler32-zlib) [ 2100.762796] Lustre: Lustre: Build Version: 2.17.58_63_g006d100 [ 2101.000459] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 2102.663871] Key type lgssc registered [ 2103.751154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2111.306647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2111.330032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2112.745506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2116.973887] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2122.224880] LustreError: 79675:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2122.260240] LustreError: 79675:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2125.810810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2126.624947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2131.367880] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2139.012782] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2145.048918] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2152.336803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2152.969384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2159.562693] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2167.521258] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2177.548574] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2178.361643] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2180.102891] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2180.108574] Lustre: Skipped 1 previous similar message [ 2182.671259] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2182.674613] Lustre: Skipped 2 previous similar messages [ 2187.465658] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2187.474650] Lustre: Skipped 4 previous similar messages [ 2195.734797] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2195.739132] Lustre: Skipped 8 previous similar messages [ 2212.403655] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2212.412581] Lustre: Skipped 17 previous similar messages [ 2244.552882] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2244.557438] Lustre: Skipped 37 previous similar messages [ 2271.354228] Lustre: 87773:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2274.119517] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 22:35:04 (1790908504) [ 2275.615879] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2277.302570] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 22:35:07 (1790908507) [ 2284.000188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2284.009047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.026668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2284.033838] Lustre: Skipped 1 previous similar message [ 2287.530345] Lustre: server umount lustre-OST0000 complete [ 2294.240741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2294.255613] Lustre: Skipped 1 previous similar message [ 2294.266586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2294.273380] Lustre: Skipped 1 previous similar message [ 2296.762436] Lustre: server umount lustre-MDT0000 complete [ 2299.360469] LustreError: 79676:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2299.368278] LustreError: 79676:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2299.737456] LustreError: 79658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790908530 with bad export cookie 11984760935193033167 [ 2299.742829] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2299.752653] LustreError: 79658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2299.982878] Lustre: server umount lustre-MDT0001 complete [ 2308.517923] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 2310.700172] Key type lgssc unregistered [ 2311.010248] LNet: 89022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2311.017466] LNetError: 89022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2311.036851] LNet: Removed LNI 192.168.203.153@tcp [ 2311.794606] Key type .llcrypt unregistered [ 2311.798720] Key type ._llcrypt unregistered [ 2315.430893] Key type ._llcrypt registered [ 2315.436234] Key type .llcrypt registered [ 2315.510410] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2319.622038] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules [ 2320.734288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2320.879432] alg: No test for adler32 (adler32-zlib) [ 2321.929818] Lustre: Lustre: Build Version: 2.17.58_63_g006d100 [ 2322.208429] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 2323.879352] Key type lgssc registered [ 2325.038791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2326.048714] LustreError: 90245:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2340.650432] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2347.787471] LustreError: 90245:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2348.102520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2348.135120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2349.602773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2353.568577] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2359.778708] LustreError: 91269:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2359.801770] LustreError: 91269:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2360.438313] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 22:36:30 (1790908590) [ 2364.738128] LustreError: 91300:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2364.747210] LustreError: 91300:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2364.759329] LustreError: 91300:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 2365.111641] Lustre: server umount lustre-MDT0000 complete [ 2376.736877] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 2379.073600] Key type lgssc unregistered [ 2379.341598] LNet: 92792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2379.348514] LNetError: 92792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2379.377639] LNet: Removed LNI 192.168.203.153@tcp [ 2380.095638] Key type .llcrypt unregistered [ 2380.101110] Key type ._llcrypt unregistered [ 2396.760990] Key type ._llcrypt registered [ 2396.764811] Key type .llcrypt registered [ 2396.889371] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2397.618149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2397.633892] alg: No test for adler32 (adler32-zlib) [ 2398.704662] Lustre: Lustre: Build Version: 2.17.58_63_g006d100 [ 2398.870705] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 2400.623619] Key type lgssc registered [ 2403.136073] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2413.088293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2413.110379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2414.547477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2419.036607] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2424.803720] LustreError: 94025:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2424.822435] LustreError: 94025:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2427.439217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2428.240179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2433.150650] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2441.306713] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2447.054947] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2453.321457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2454.099371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2460.821863] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2467.635285] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2516.510591] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 22:39:06 (1790908746) [ 2518.384709] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2520.549168] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 22:39:10 (1790908750) [ 2525.152223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2525.158637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2525.176775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2529.250771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2529.252301] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2529.272090] Lustre: Skipped 1 previous similar message [ 2530.742348] Lustre: server umount lustre-MDT0000 complete [ 2534.030354] LustreError: 94005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790908765 with bad export cookie 13896737816219851324 [ 2534.033133] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2534.048530] LustreError: 94005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2534.278945] Lustre: server umount lustre-MDT0001 complete [ 2547.421362] Lustre: server umount lustre-OST0000 complete [ 2559.277287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.749154] LustreError: 98303:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.963317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2565.074629] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2565.099485] LustreError: 98304:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2569.187205] LustreError: 98303:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.301673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2573.838936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2578.002495] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2584.985297] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2590.292526] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2603.774224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2604.039662] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2604.050626] LustreError: 98292:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2604.099064] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2604.111872] LustreError: 100387:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2604.123580] LustreError: 100387:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2604.138781] LustreError: 100387:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2604.232616] Lustre: server umount lustre-OST0000 complete [ 2604.235694] LustreError: 100387:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2614.049684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2614.192581] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2614.206756] LustreError: 98292:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2614.242834] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2614.252288] LustreError: 100855:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2614.256199] LustreError: 100855:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2614.263349] LustreError: 100855:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2614.300920] Lustre: server umount lustre-OST0000 complete [ 2614.304577] LustreError: 100855:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2626.377745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2626.627078] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2626.636751] Lustre: Found index 0 for lustre-OST0000, updating log [ 2626.649930] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2626.735992] Lustre: lustre-OST0000: new disk, initializing [ 2626.740564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2626.749984] Lustre: 101393:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2627.076387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2633.686171] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2637.249491] Lustre: Failing over lustre-OST0000 [ 2637.400822] Lustre: server umount lustre-OST0000 complete [ 2643.237713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2643.249132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2643.271842] Lustre: Skipped 1 previous similar message [ 2643.287675] Lustre: Skipped 3 previous similar messages [ 2648.544319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2648.557538] Lustre: Skipped 1 previous similar message [ 2651.697615] Lustre: server umount lustre-MDT0000 complete [ 2653.666685] LustreError: 98303:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.685299] LustreError: 98303:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2655.208161] LustreError: 98284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790908886 with bad export cookie 13896737816219854068 [ 2655.225654] LustreError: 98284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2655.228453] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2655.624358] Lustre: server umount lustre-MDT0001 complete [ 2664.421269] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 22:41:34 (1790908894) [ 2679.090097] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2689.607598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2689.989455] LustreError: 103733:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2690.172885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2694.652656] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2703.844449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2708.863484] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2711.502913] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2717.840664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2718.397487] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2723.971233] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2729.641780] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2729.654547] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2729.676180] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2732.585647] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2732.671057] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2732.744480] Lustre: lustre-OST0001: new disk, initializing [ 2732.747937] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2732.756289] Lustre: 106051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2733.048340] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:56 [ 2733.072439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2733.090453] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2733.093423] Lustre: Skipped 2 previous similar messages [ 2735.151168] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:04 [ 2735.164860] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2735.165790] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2735.171358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2735.176932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2739.827462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2739.834746] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2739.844909] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2740.028150] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2748.313900] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2755.622246] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2760.684136] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2760.696878] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2760.706227] Lustre: Skipped 1 previous similar message [ 2760.715506] Lustre: Skipped 2 previous similar messages [ 2765.978033] Lustre: server umount lustre-MDT0001 complete [ 2768.438925] LustreError: 103730:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2768.468599] LustreError: 103730:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2775.990429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2776.297753] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2776.306340] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2776.435441] Lustre: 103721:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2778.430117] LustreError: 107603:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2778.440344] LustreError: 107603:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2778.456765] LustreError: 103725:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2778.636437] LustreError: 107619:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.153@tcp: cfg command failed: rc = -17 [ 2778.654370] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2778.654370] [ 2782.003470] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2782.705337] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2791.496928] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 22:43:41 (1790909021) [ 2807.714545] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2824.060260] Lustre: 109667:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2844.389537] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2847.832086] Lustre: 110803:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2930.656952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2930.662181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2930.673973] Lustre: Skipped 1 previous similar message [ 2930.679296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2930.685547] Lustre: Skipped 4 previous similar messages [ 2935.704817] Lustre: server umount lustre-MDT0000 complete [ 2936.287894] LustreError: 103729:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2936.308626] LustreError: 103729:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 2938.585481] LustreError: 103714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790909169 with bad export cookie 13896737816219855069 [ 2938.591076] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2938.598625] LustreError: 103714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2938.931436] Lustre: server umount lustre-MDT0001 complete [ 2952.749652] Lustre: server umount lustre-OST0000 complete [ 2965.198268] Lustre: server umount lustre-OST0001 complete [ 2970.200785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2980.196869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2987.673795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2997.480856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3017.835249] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 3026.140570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3026.159500] Lustre: lustre-MDT0000: reset Object Index mappings [ 3026.437576] LustreError: 114756:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3026.515448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3026.524180] Lustre: Skipped 1 previous similar message [ 3029.945878] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3037.452254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3037.468961] Lustre: lustre-MDT0001: reset Object Index mappings [ 3041.804543] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3044.233814] Lustre: 115906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3050.084210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3051.838793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:382 to 0x280000401:418) [ 3057.084453] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3062.265784] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3065.639711] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3071.491655] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3071.501041] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3073.152838] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3081.592104] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3085.879192] Lustre: 117781:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3138.492412] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 22:49:28 (1790909368) [ 3166.177591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3166.185826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3166.201343] Lustre: Skipped 3 previous similar messages [ 3166.217045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3166.226126] Lustre: Skipped 4 previous similar messages [ 3171.505567] Lustre: server umount lustre-MDT0000 complete [ 3173.867716] LustreError: 115492:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3173.895707] LustreError: 115492:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 3175.694105] LustreError: 114737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790909406 with bad export cookie 13896737816219902165 [ 3175.711794] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3190.239623] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3190.739545] Lustre: server umount lustre-MDT0001 complete [ 3202.417188] Lustre: server umount lustre-OST0000 complete [ 3212.152990] Lustre: server umount lustre-OST0001 complete [ 3218.848581] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 3230.156606] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 3262.841422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3263.097374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3263.139764] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3263.247427] Lustre: lustre-MDT0000: new disk, initializing [ 3263.335394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3263.338406] Lustre: Skipped 3 previous similar messages [ 3263.360322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3267.527343] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3273.884534] Lustre: server umount lustre-MDT0000 complete [ 3282.051883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3282.444610] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3282.458963] Lustre: 122060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3282.469430] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3287.105853] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3292.531399] Lustre: server umount lustre-MDT0000 complete [ 3301.600077] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 22:52:11 (1790909531) [ 3303.320302] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3305.458243] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 22:52:15 (1790909535) [ 3324.518318] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 3333.286020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3337.518517] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3345.855543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3346.043529] Lustre: 124110:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3346.124955] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3346.129844] Lustre: Skipped 1 previous similar message [ 3346.193214] Lustre: lustre-MDT0001: new disk, initializing [ 3346.300157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3346.315475] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3350.095231] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3354.789504] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3361.531272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3361.771962] Lustre: lustre-OST0000: new disk, initializing [ 3361.775611] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3361.781244] Lustre: 125689:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3363.638783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3363.655101] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3363.756399] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3367.626419] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3378.655343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3378.815418] Lustre: lustre-OST0001: new disk, initializing [ 3378.818957] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3378.824528] Lustre: 126559:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3380.217062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3380.227498] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3380.372816] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3385.691404] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3395.777991] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3399.370442] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3779.466629] Lustre: 132976:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3779.479549] Lustre: 132976:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3779.486570] Lustre: 132976:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3779.498690] Lustre: 132976:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3779.504262] Lustre: 132976:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3779.509600] Lustre: 132976:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4081.603630] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:05:10 (1790910310) [ 4096.997877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4097.007164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4097.013765] Lustre: Skipped 8 previous similar messages [ 4097.033630] Lustre: Skipped 15 previous similar messages [ 4102.112965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4102.132361] Lustre: Skipped 3 previous similar messages [ 4102.723694] Lustre: server umount lustre-MDT0000 complete [ 4106.331795] LustreError: 125699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790910337 with bad export cookie 13896737816219971549 [ 4106.339197] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4106.345254] LustreError: 125699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4107.233697] LustreError: 124965:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4107.235086] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4107.253312] LustreError: 124965:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 4107.275517] Lustre: Skipped 1 previous similar message [ 4111.275893] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4111.282354] Lustre: Skipped 3 previous similar messages [ 4112.807222] Lustre: server umount lustre-MDT0001 complete [ 4122.590986] Lustre: server umount lustre-OST0000 complete [ 4132.293785] Lustre: server umount lustre-OST0001 complete [ 4139.390828] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 4150.731306] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 4187.100119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4187.340216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4187.378725] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4187.463592] Lustre: lustre-MDT0000: new disk, initializing [ 4187.565288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4187.581928] Lustre: Skipped 5 previous similar messages [ 4187.603940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4192.556986] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4203.225138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4203.357679] Lustre: 136039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4203.415144] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4203.419673] Lustre: Skipped 1 previous similar message [ 4203.516265] Lustre: lustre-MDT0001: new disk, initializing [ 4203.677222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4203.686095] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4207.835603] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4218.431363] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4224.736068] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4232.155186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4232.505639] Lustre: lustre-OST0000: new disk, initializing [ 4232.509608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4232.519976] Lustre: 138009:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4232.612955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4232.619823] Lustre: Skipped 1 previous similar message [ 4234.132139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4234.164612] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4234.298420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4238.743127] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4248.402462] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4257.687207] Lustre: lustre-MDD0000: changelog on [ 4260.305749] Lustre: lustre-MDD0001: changelog on [ 6320.333586] Lustre: lustre-MDD0001: changelog off [ 6323.012325] Lustre: 138928:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6324.121452] Lustre: lustre-MDD0000: changelog off [ 6333.411749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6333.423542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6333.449805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6337.651700] Lustre: server umount lustre-OST0000 complete [ 6343.654969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6343.668841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6343.677395] Lustre: Skipped 1 previous similar message [ 6343.683387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6343.688722] Lustre: Skipped 2 previous similar messages [ 6348.780776] LustreError: 136052:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6348.799354] LustreError: 136052:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6348.871289] Lustre: server umount lustre-MDT0000 complete [ 6352.828585] LustreError: 136031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790912583 with bad export cookie 13896737816220791739 [ 6352.837633] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6352.845615] LustreError: 136031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6353.228953] Lustre: server umount lustre-MDT0001 complete [ 6364.477922] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 6367.325675] Key type lgssc unregistered [ 6367.662520] LNet: 147173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6367.668672] LNetError: 147173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6367.678406] LNet: Removed LNI 192.168.203.153@tcp [ 6368.667203] Key type .llcrypt unregistered [ 6368.669765] Key type ._llcrypt unregistered [ 6388.771738] Key type ._llcrypt registered [ 6388.774269] Key type .llcrypt registered [ 6388.893414] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 6402.885215] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6403.935299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6404.034801] alg: No test for adler32 (adler32-zlib) [ 6405.063910] Lustre: Lustre: Build Version: 2.17.58_63_g006d100 [ 6405.370193] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 6407.040910] Key type lgssc registered [ 6408.411233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6449.743930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6449.792192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6451.103539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6451.121273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6451.247384] Lustre: lustre-MDT0000: new disk, initializing [ 6451.363600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6451.390521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6456.109821] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6466.278962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6466.376397] Lustre: 150390:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6466.419711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6466.423621] Lustre: Skipped 1 previous similar message [ 6466.489408] Lustre: lustre-MDT0001: new disk, initializing [ 6466.566432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6466.603135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6466.618214] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6470.948918] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6480.061447] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6485.582924] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6491.959351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6492.180692] Lustre: lustre-OST0000: new disk, initializing [ 6492.187596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6492.196586] Lustre: 152357:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6492.285785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6498.294775] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6498.384573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6498.395867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6498.505333] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6506.837695] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6512.839705] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_module obdecho/obdecho [ 6514.698624] LustreError: 153359:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6514.717312] LustreError: 153359:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6516.502656] LustreError: 153455:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6529.514518] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6529.518359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6529.541932] Lustre: Skipped 1 previous similar message [ 6534.635451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6534.640246] Lustre: Skipped 3 previous similar messages [ 6535.265183] Lustre: server umount lustre-MDT0000 complete [ 6539.471731] LustreError: 150382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790912770 with bad export cookie 5425929478198911829 [ 6539.473869] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6539.500532] LustreError: 150382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6539.745749] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6539.762617] Lustre: Skipped 1 previous similar message [ 6539.986621] Lustre: server umount lustre-MDT0001 complete [ 6546.336171] Lustre: server umount lustre-OST0000 complete [ 6568.981738] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 6579.609129] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6616.483687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6616.847847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6616.877330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6617.005349] Lustre: lustre-MDT0000: new disk, initializing [ 6617.127060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6617.155459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6622.379305] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6633.551565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6633.682799] Lustre: 157170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6633.746569] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6633.751467] Lustre: Skipped 1 previous similar message [ 6633.835746] Lustre: lustre-MDT0001: new disk, initializing [ 6633.935363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6633.975370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6633.989366] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6638.404348] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6648.180066] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6654.108351] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6660.896243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6661.158540] Lustre: lustre-OST0000: new disk, initializing [ 6661.162516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6661.171698] Lustre: 159139:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6661.259833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6666.482591] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6671.410085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6671.422489] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6671.516598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6675.434244] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6680.416738] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6680.674487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6686.204715] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6686.539717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6692.329627] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6692.337912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6692.340349] LustreError: Skipped 1 previous similar message [ 6692.341627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6692.356542] Lustre: Skipped 1 previous similar message [ 6692.378457] Lustre: Skipped 1 previous similar message [ 6694.949486] Lustre: server umount lustre-OST0000 complete [ 6700.520306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6700.527987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6700.557299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6707.681444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6707.685295] Lustre: Skipped 3 previous similar messages [ 6713.311116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6713.608298] Lustre: server umount lustre-MDT0000 complete [ 6717.604686] LustreError: 157163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790912948 with bad export cookie 5425929478198912795 [ 6717.612278] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6717.627181] LustreError: 157163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6717.924182] Lustre: server umount lustre-MDT0001 complete [ 6721.615052] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 23:49:11 (1790912951) [ 6723.284276] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6724.985693] Lustre: DEBUG MARKER: == conf-sanity test 138: MDT mount failure must not LBUG destroying recovery exports ========================================================== 23:49:15 (1790912955) [ 6743.225870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6743.576153] LustreError: 162115:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6743.592710] LustreError: 162115:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6743.635907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6748.557655] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6748.642748] LustreError: 162116:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6753.761684] LustreError: 162115:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6756.914165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6761.660938] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6769.936136] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6775.962263] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6783.596059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6783.879121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6783.887375] Lustre: Skipped 1 previous similar message [ 6789.642784] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6796.974532] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6805.450574] Lustre: *** cfs_fail_loc=718, val=0*** [ 6805.684755] Lustre: server umount lustre-MDT0000 complete [ 6807.525366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6807.532941] LustreError: 162112:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6807.553083] Lustre: Skipped 2 previous similar messages [ 6807.587382] LustreError: 162112:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6808.953001] LustreError: 162095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790913039 with bad export cookie 5425929478198913677 [ 6808.955324] Lustre: *** cfs_fail_loc=718, val=0*** [ 6808.957329] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6808.963780] LustreError: 162095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6808.969463] Lustre: Skipped 3 previous similar messages [ 6811.410329] Lustre: server umount lustre-MDT0001 complete [ 6820.762803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6828.959232] Lustre: 148606:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790913043/real 1790913043] req@00000000ff7868b6 x1877908008316160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790913059 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6828.985420] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6828.993317] Lustre: Skipped 1 previous similar message [ 6833.119918] Lustre: 148603:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790913048/real 1790913048] req@00000000477f971c x1877908008316416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790913064 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6833.121107] LustreError: 148602:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000021d13a38 x1877908008317312/t0(0) o250->MGC192.168.203.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6833.140514] Lustre: 148603:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6833.399554] Lustre: *** cfs_fail_loc=723, val=0*** [ 6833.401961] Lustre: Skipped 1 previous similar message [ 6833.474458] LustreError: 165468:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-122) [ 6833.483461] LustreError: 165468:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.153@tcp: cfg command failed: rc = -122 [ 6833.492529] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6833.492529] [ 6833.504867] LustreError: MGC192.168.203.153@tcp: Confguration from log lustre-MDT0000 failed from MGS -122. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6833.521894] LustreError: 165439:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -122 [ 6833.529492] LustreError: 165439:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -122 [ 6833.539150] LustreError: 165439:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 6833.548618] LustreError: 165439:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6833.649222] Lustre: server umount lustre-MDT0000 complete [ 6833.654078] LustreError: 165439:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -122 [ 6834.591119] Lustre: 148606:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790913049/real 1790913049] req@000000008a5e10a9 x1877908008316672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790913065 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6839.711106] Lustre: 148605:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790913054/real 1790913054] req@00000000eb4a2a6f x1877908008316800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790913070 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6841.116320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6841.291948] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6841.582317] LustreError: 163945:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6841.604103] LustreError: 163945:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6843.743245] Lustre: 148604:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790913058/real 1790913058] req@0000000058d0ca21 x1877908008317056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790913074 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6846.805442] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6847.798142] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6853.092685] LustreError: 163946:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6853.108982] LustreError: 163946:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6855.334324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6855.802190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6855.811198] Lustre: Skipped 1 previous similar message [ 6855.866441] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6857.569407] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6859.987830] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6861.293487] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6861.306538] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6867.642302] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6869.230287] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6875.766056] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6877.622872] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6887.904497] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6887.922139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6887.939195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6887.943398] Lustre: Skipped 2 previous similar messages [ 6892.014957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6893.944182] Lustre: server umount lustre-OST0000 complete [ 6904.159061] Lustre: server umount lustre-MDT0000 complete [ 6905.313668] LustreError: 165876:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6905.346702] LustreError: 165876:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6908.329212] LustreError: 162095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790913139 with bad export cookie 5425929478198914622 [ 6908.342034] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6908.351959] LustreError: 162095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6908.863148] Lustre: server umount lustre-MDT0001 complete [ 6920.032798] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 6922.829799] Key type lgssc unregistered [ 6923.108484] LNet: 168697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6923.120721] LNetError: 168697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6923.146880] LNet: Removed LNI 192.168.203.153@tcp [ 6923.935703] Key type .llcrypt unregistered [ 6923.939375] Key type ._llcrypt unregistered [ 6943.700706] Key type ._llcrypt registered [ 6943.703233] Key type .llcrypt registered [ 6943.809628] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6944.776127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6944.815220] alg: No test for adler32 (adler32-zlib) [ 6945.812684] Lustre: Lustre: Build Version: 2.17.58_63_g006d100 [ 6946.086698] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 6947.738830] Key type lgssc registered [ 6949.059087] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6956.053143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6956.077636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6957.539510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6961.274347] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6967.778135] LustreError: 170030:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6967.793402] LustreError: 170030:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6968.811048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6969.411258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6973.817674] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6979.900739] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6984.855656] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6991.641133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6991.980775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6998.279689] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7002.243202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7006.028271] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7012.854469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7012.858074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7012.864570] Lustre: Skipped 2 previous similar messages [ 7012.876516] Lustre: Skipped 2 previous similar messages [ 7015.102060] Lustre: server umount lustre-MDT0000 complete [ 7017.953149] LustreError: 170756:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7017.974551] LustreError: 170756:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7018.387762] LustreError: 170010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790913249 with bad export cookie 4104176788118201665 [ 7018.396152] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7018.401313] LustreError: 170010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7023.078773] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7023.079158] LustreError: 172376:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7023.091932] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7023.102253] LustreError: 172376:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7023.118373] Lustre: Skipped 1 previous similar message [ 7024.782375] Lustre: server umount lustre-MDT0001 complete [ 7030.505077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7043.279757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7043.526595] LustreError: 173871:0:(llog.c:1655:llog_backup()) MGC192.168.203.153@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7043.534206] Lustre: 173871:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.153@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7043.552723] LustreError: 173876:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7043.569209] LustreError: 173876:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7048.672623] LustreError: 173878:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7057.967167] LustreError: 173876:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7057.979644] LustreError: 173876:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7074.350308] LustreError: 173877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7074.386441] LustreError: 173877:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 16 previous similar messages [ 7104.479550] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7104.486563] Lustre: 173871:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.153@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7109.096842] LustreError: 171858:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7109.112091] LustreError: 171858:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 23 previous similar messages [ 7145.464396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7150.416885] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7150.570174] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7159.116992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7163.875831] LustreError: 169327:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000032b97b2d x1877908575531392/t0(0) o250->MGC192.168.203.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7164.299605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7164.400934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 7169.125279] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7169.518173] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7176.480520] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7178.496272] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7184.383422] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7186.196473] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7205.346924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7205.350730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7205.354690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7205.354698] Lustre: Skipped 1 previous similar message [ 7205.393719] Lustre: Skipped 2 previous similar messages [ 7210.374397] Lustre: server umount lustre-MDT0000 complete [ 7210.471078] LustreError: 173878:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7210.486379] LustreError: 173878:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 42 previous similar messages [ 7215.145520] LustreError: 170009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790913446 with bad export cookie 4104176788118208336 [ 7215.148978] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7215.154610] LustreError: 170009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7215.588403] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7215.735764] Lustre: server umount lustre-MDT0001 complete [ 7222.203487] Lustre: server umount lustre-OST0000 complete [ 7245.763135] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 7255.531644] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 7295.192160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7295.459826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7295.488226] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7295.603629] Lustre: lustre-MDT0000: new disk, initializing [ 7295.722664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7295.753164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7300.989105] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7311.646197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7311.742716] Lustre: 178974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7311.795841] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7311.799847] Lustre: Skipped 1 previous similar message [ 7311.852867] Lustre: lustre-MDT0001: new disk, initializing [ 7311.928103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7311.967514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7311.979880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7316.942995] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7326.195192] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7332.765530] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7341.285675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7341.810538] Lustre: lustre-OST0000: new disk, initializing [ 7341.814443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7341.825560] Lustre: 180941:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7347.826684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7347.839624] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7348.015411] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7350.511523] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7358.505606] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7363.237695] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7363.411376] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7367.791543] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7368.015493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7373.807364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7373.817435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7373.823766] Lustre: Skipped 1 previous similar message [ 7373.825188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7373.855984] Lustre: Skipped 3 previous similar messages [ 7376.309893] Lustre: server umount lustre-OST0000 complete [ 7383.522269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7383.542274] LustreError: Skipped 1 previous similar message [ 7383.560478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7383.570692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7386.401366] Lustre: server umount lustre-MDT0000 complete [ 7389.152269] LustreError: 178986:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7389.171300] LustreError: 178986:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7389.774585] LustreError: 178967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790913620 with bad export cookie 4104176788118225388 [ 7389.782152] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7389.798303] LustreError: 178967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7390.248335] Lustre: server umount lustre-MDT0001 complete [ 7399.058922] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 00:00:29 (1790913629) [ 7404.726099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7405.068124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7405.079151] Lustre: Skipped 1 previous similar message [ 7409.057185] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7416.810375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7421.755171] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7429.493411] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7435.856642] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7442.173033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7448.528572] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7455.578656] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7465.014117] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 00:01:35 (1790913695) [ 7469.550302] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7469.567789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7469.581725] Lustre: Skipped 1 previous similar message [ 7469.596573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7469.607812] Lustre: Skipped 2 previous similar messages [ 7474.774434] Lustre: server umount lustre-OST0000 complete [ 7478.753819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7478.764562] LustreError: Skipped 1 previous similar message [ 7484.741778] Lustre: server umount lustre-MDT0000 complete [ 7488.502114] LustreError: 183336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790913719 with bad export cookie 4104176788118226277 [ 7488.511994] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7488.516470] LustreError: 183336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7488.973559] Lustre: server umount lustre-MDT0001 complete [ 7500.289683] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 7503.221596] Key type lgssc unregistered [ 7503.475509] LNet: 187095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7503.482786] LNetError: 187095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7503.514530] LNet: Removed LNI 192.168.203.153@tcp [ 7504.393087] Key type .llcrypt unregistered [ 7504.396251] Key type ._llcrypt unregistered [ 7522.415930] Key type ._llcrypt registered [ 7522.418680] Key type .llcrypt registered [ 7522.534717] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 7523.562115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7523.793909] alg: No test for adler32 (adler32-zlib) [ 7525.026907] Lustre: Lustre: Build Version: 2.17.58_63_g006d100 [ 7525.364909] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 7527.079191] Key type lgssc registered [ 7528.321286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7537.364143] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7537.392210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7552.991530] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7558.111426] LustreError: 188353:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.153@tcp: failed processing log, type 4: rc = -110 [ 7563.232407] LustreError: 188353:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7563.249420] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7563.258350] LustreError: MGC192.168.203.153@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7563.279545] LustreError: 188353:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7563.293342] LustreError: 188353:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 7563.309941] LustreError: 188353:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 7563.353574] Lustre: server umount lustre-OST0000 complete [ 7563.360737] LustreError: 188353:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7571.602037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7572.979937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7577.685379] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7582.181585] LustreError: 188854:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7582.205463] LustreError: 188854:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7586.196542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7586.529912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7591.224821] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7598.804021] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7604.791717] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7612.419942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7612.662506] LustreError: 190669:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7612.669676] Lustre: 190669:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.153@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7612.784680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7619.915333] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7628.258249] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7631.596761] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7631.599780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7631.619484] Lustre: Skipped 2 previous similar messages [ 7631.641333] Lustre: Skipped 2 previous similar messages [ 7636.963995] LustreError: 188851:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7636.992894] LustreError: 188851:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7637.130547] Lustre: server umount lustre-MDT0000 complete [ 7641.400366] LustreError: 188836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790913872 with bad export cookie 16798482374540261226 [ 7641.406866] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7641.848610] LustreError: 189583:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000dabc30cf x1877909182639744/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7642.019448] Lustre: server umount lustre-MDT0001 complete [ 7655.379420] Lustre: server umount lustre-OST0000 complete [ 7665.465389] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 7677.529854] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 7717.646795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7718.090532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7718.146820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7718.319432] Lustre: lustre-MDT0000: new disk, initializing [ 7718.443279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7718.484321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7725.383832] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7736.863807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7736.964340] Lustre: 194195:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7737.009578] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7737.014436] Lustre: Skipped 1 previous similar message [ 7737.077931] Lustre: lustre-MDT0001: new disk, initializing [ 7737.152563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7737.192792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7737.203882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7741.398650] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7749.593649] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7754.896390] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7761.361439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7761.583271] Lustre: lustre-OST0000: new disk, initializing [ 7761.586597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7761.594768] Lustre: 196163:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7761.673177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7767.133485] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7768.137149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7768.152753] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7768.264395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7775.904164] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7780.054045] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7780.263747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7784.285063] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7784.449880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7789.024203] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7789.024887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7789.040995] LustreError: Skipped 1 previous similar message [ 7789.042625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7789.079377] Lustre: Skipped 1 previous similar message [ 7792.282311] Lustre: server umount lustre-OST0000 complete [ 7798.759296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7798.769479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7798.797765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7798.800685] Lustre: Skipped 1 previous similar message [ 7802.075362] Lustre: server umount lustre-MDT0000 complete [ 7804.386491] LustreError: 194208:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7804.413796] LustreError: 194208:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7805.734430] LustreError: 196162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790914036 with bad export cookie 16798482374540261954 [ 7805.736806] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7805.740286] LustreError: 196162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7806.080938] Lustre: server umount lustre-MDT0001 complete [ 7814.565411] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 00:07:24 (1790914044) [ 7830.794445] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 7833.662600] Key type lgssc unregistered [ 7833.953127] LNet: 199104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7833.960355] LNetError: 199104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7834.988176] LNet: Removed LNI 192.168.203.153@tcp [ 7835.977464] Key type .llcrypt unregistered [ 7835.979590] Key type ._llcrypt unregistered [ 7855.998524] Key type ._llcrypt registered [ 7856.001403] Key type .llcrypt registered [ 7856.126521] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 7857.443703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7857.615610] alg: No test for adler32 (adler32-zlib) [ 7858.799769] Lustre: Lustre: Build Version: 2.17.58_63_g006d100 [ 7859.140850] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 7860.969076] Key type lgssc registered [ 7862.389294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7871.080904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7871.109334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7872.640188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7876.946426] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7882.221924] LustreError: 200338:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7882.236205] LustreError: 200338:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7886.332789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7886.876266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7891.472727] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7898.442130] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7904.272899] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7912.759459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7912.975371] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7912.978899] LustreError: 202203:0:(llog.c:1696:llog_backup()) MGC192.168.203.153@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7912.985470] Lustre: 202203:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.153@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7913.117331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7919.925302] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7928.319416] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7931.375991] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7931.380825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7931.410982] Lustre: Skipped 2 previous similar messages [ 7931.433654] Lustre: Skipped 2 previous similar messages [ 7936.488716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7936.500808] Lustre: Skipped 2 previous similar messages [ 7936.591982] Lustre: server umount lustre-MDT0000 complete [ 7940.357137] LustreError: 200318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790914171 with bad export cookie 842915274010029954 [ 7940.362439] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7940.373956] LustreError: 200318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7941.605588] LustreError: 202228:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7941.606672] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7941.649625] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7941.727315] LustreError: 201068:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003ed7e3aa x1877909532863872/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7941.941098] Lustre: server umount lustre-MDT0001 complete [ 7948.196790] Lustre: server umount lustre-OST0000 complete [ 7957.516959] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 7969.087213] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 8004.558459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8004.834767] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8004.872981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8005.012139] Lustre: lustre-MDT0000: new disk, initializing [ 8005.177457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8005.213279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8009.441073] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8019.597597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8019.671946] Lustre: 205728:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8019.715631] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8019.717734] Lustre: Skipped 1 previous similar message [ 8019.720485] LustreError: 206511:0:(llog.c:1696:llog_backup()) MGC192.168.203.153@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8019.727628] LustreError: 206511:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8019.734738] Lustre: 206511:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.153@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8019.754834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8019.758677] Lustre: Skipped 1 previous similar message [ 8019.835604] Lustre: lustre-MDT0001: new disk, initializing [ 8019.915040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8019.961794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8019.980746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8024.346959] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8033.285474] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8039.782442] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8047.038130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8047.293275] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8047.295980] Lustre: Skipped 1 previous similar message [ 8047.299658] LustreError: 207673:0:(llog.c:1696:llog_backup()) MGC192.168.203.153@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8047.309459] LustreError: 207673:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8047.315938] Lustre: 207673:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.153@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8047.328879] Lustre: 207673:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8047.350520] Lustre: lustre-OST0000: new disk, initializing [ 8047.354901] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8047.362727] Lustre: 207696:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8047.490239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8053.334606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8053.348870] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8053.495586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8053.981864] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8063.214285] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8067.588578] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8067.823873] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8072.377301] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8072.586833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8079.329585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8079.348287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8079.362762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8079.371143] Lustre: Skipped 1 previous similar message [ 8080.474978] Lustre: server umount lustre-OST0000 complete [ 8084.449108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8084.452107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8084.457207] Lustre: Skipped 2 previous similar messages [ 8084.466111] Lustre: Skipped 1 previous similar message [ 8090.284987] Lustre: server umount lustre-MDT0000 complete [ 8093.853178] LustreError: 205721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790914324 with bad export cookie 842915274010030668 [ 8093.854484] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8093.862378] LustreError: 205721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8094.232589] Lustre: server umount lustre-MDT0001 complete [ 8102.241301] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 00:12:12 (1790914332) [ 8119.455212] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 8128.783791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8129.198245] LustreError: 210821:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8129.241527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8133.337849] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8134.631543] LustreError: 210822:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8139.762783] LustreError: 210821:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8142.070756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8147.045549] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8149.943448] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8156.396587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8162.757573] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8171.574827] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8171.781828] Lustre: lustre-OST0001: new disk, initializing [ 8171.786206] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8171.794294] Lustre: 213130:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8171.930882] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8171.942255] Lustre: Skipped 2 previous similar messages [ 8178.284929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8178.301196] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8178.412954] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8178.509494] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8190.149952] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8199.330249] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8201.355925] Lustre: DEBUG MARKER: ADD OST3 [ 8206.775839] loop: module loaded [ 8206.846391] loop0: detected capacity change from 0 to 204800000 [ 8207.153377] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8207.166255] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8207.185873] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8214.430584] loop0: detected capacity change from 0 to 204800000 [ 8214.448910] Lustre: DEBUG MARKER: STOP OST3 [ 8221.037360] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8221.307230] Lustre: lustre-OST0003: new disk, initializing [ 8221.321239] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8221.334325] Lustre: 215178:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8222.745368] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8222.930784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8222.944973] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8222.986302] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8224.119626] Lustre: Failing over lustre-OST0003 [ 8224.321733] Lustre: server umount lustre-OST0003 complete [ 8224.799614] LustreError: 215297:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8227.938164] LustreError: 213479:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8228.323800] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8228.698263] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8239.669566] Lustre: DEBUG MARKER: START OST3 again [ 8246.795292] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8247.138530] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8247.149536] Lustre: Skipped 1 previous similar message [ 8247.179419] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8248.399969] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8248.713254] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8248.713356] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8253.717973] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8275.693204] Lustre: Failing over lustre-OST0003 [ 8275.817813] Lustre: server umount lustre-OST0003 complete [ 8278.007385] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8278.029931] LustreError: 212307:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8278.031037] Lustre: Skipped 2 previous similar messages [ 8278.057028] LustreError: 212307:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 8295.914484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8295.928807] LustreError: Skipped 1 previous similar message [ 8295.932739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8295.963909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8295.973869] Lustre: Skipped 2 previous similar messages [ 8299.901521] Lustre: server umount lustre-MDT0000 complete [ 8303.578663] LustreError: 210802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790914534 with bad export cookie 842915274010031557 [ 8303.580481] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8303.595737] LustreError: 210802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8310.054711] Lustre: server umount lustre-MDT0001 complete [ 8314.216343] Lustre: server umount lustre-OST0000 complete [ 8318.127941] Lustre: server umount lustre-OST0001 complete [ 8325.831095] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 8334.442889] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 8372.622713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8372.904786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8372.952275] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8373.056942] Lustre: lustre-MDT0000: new disk, initializing [ 8373.183250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8377.871541] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8389.467361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8389.607333] Lustre: 220744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8389.662974] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8389.667627] Lustre: Skipped 1 previous similar message [ 8389.756935] Lustre: lustre-MDT0001: new disk, initializing [ 8389.847144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8389.857272] Lustre: Skipped 1 previous similar message [ 8389.880672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8389.886674] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8394.464373] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8404.435313] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8411.324164] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8418.559921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8418.874865] Lustre: 222713:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8420.574255] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8420.729895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8425.493492] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8435.732263] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8440.722374] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8440.915393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8445.358324] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8445.572714] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8451.557396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8451.559329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8451.567319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8451.573278] Lustre: Skipped 6 previous similar messages [ 8451.596687] Lustre: Skipped 9 previous similar messages [ 8453.668762] Lustre: server umount lustre-OST0000 complete [ 8461.794728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8461.805347] LustreError: Skipped 1 previous similar message [ 8463.071777] Lustre: server umount lustre-MDT0000 complete [ 8466.596067] LustreError: 220737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790914697 with bad export cookie 842915274010033853 [ 8466.609833] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8466.619047] LustreError: 220737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8466.916682] Lustre: server umount lustre-MDT0001 complete [ 8470.273573] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 00:18:20 (1790914700) [ 8486.582682] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 8496.520911] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 8530.124894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8530.449921] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8530.489282] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8530.493740] Lustre: Skipped 1 previous similar message [ 8530.560458] Lustre: lustre-MDT0000: new disk, initializing [ 8530.564381] Lustre: Skipped 1 previous similar message [ 8530.708683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8530.719356] Lustre: Skipped 1 previous similar message [ 8536.442256] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8547.622275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8547.829830] Lustre: 227325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8548.144338] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8554.414992] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8568.816064] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8575.259098] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8582.121748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8582.350487] Lustre: 229294:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8583.818910] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8583.974979] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8587.985932] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8596.714977] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8601.031162] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8601.227907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8605.511492] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8605.727382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8609.764952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8609.766297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8609.769851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8609.769859] Lustre: Skipped 2 previous similar messages [ 8609.773544] LustreError: Skipped 1 previous similar message [ 8609.801133] Lustre: Skipped 3 previous similar messages [ 8613.580415] Lustre: server umount lustre-OST0000 complete [ 8620.006808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8623.430261] Lustre: server umount lustre-MDT0000 complete [ 8625.127880] LustreError: 227332:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8625.143125] LustreError: 227332:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 8626.638760] LustreError: 227318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790914857 with bad export cookie 842915274010034735 [ 8626.654427] LustreError: 227318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8626.658765] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8626.977479] Lustre: server umount lustre-MDT0001 complete [ 8635.470786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8640.621847] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8649.448636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8649.845724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8649.851750] Lustre: Skipped 5 previous similar messages [ 8654.427374] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8662.235853] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8667.398906] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8674.097505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8680.824910] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8688.911807] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8701.423832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8701.426365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8701.440780] LustreError: Skipped 1 previous similar message [ 8701.445431] Lustre: Skipped 4 previous similar messages [ 8704.975208] Lustre: server umount lustre-OST0000 complete [ 8715.346309] Lustre: server umount lustre-MDT0000 complete [ 8716.770871] LustreError: 231514:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8716.784234] LustreError: 231514:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 8719.003937] LustreError: 231494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790914950 with bad export cookie 842915274010035617 [ 8719.004826] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8719.020154] LustreError: 231494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8719.298740] Lustre: server umount lustre-MDT0001 complete [ 8828.502856] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 8831.948337] Key type lgssc unregistered [ 8832.293213] LNet: 235174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8832.316031] LNetError: 235174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8832.339708] LNet: Removed LNI 192.168.203.153@tcp [ 8833.467242] Key type .llcrypt unregistered [ 8833.477092] Key type ._llcrypt unregistered [ 8858.939434] Key type ._llcrypt registered [ 8858.942751] Key type .llcrypt registered [ 8859.040833] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 8875.109591] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 8876.509878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8876.614159] alg: No test for adler32 (adler32-zlib) [ 8877.803357] Lustre: Lustre: Build Version: 2.17.58_63_g006d100 [ 8878.068788] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 8879.805768] Key type lgssc registered [ 8881.212839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8920.260083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8920.293625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8921.739582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8921.782687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8921.837097] Lustre: lustre-MDT0000: new disk, initializing [ 8921.910365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8921.929209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8927.727665] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8939.348364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8939.426384] Lustre: 238546:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8939.473947] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8939.476948] Lustre: Skipped 1 previous similar message [ 8939.615584] Lustre: lustre-MDT0001: new disk, initializing [ 8939.746759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8939.821120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8939.848148] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8944.833722] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8955.129409] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8961.843500] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8969.836196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8970.159768] Lustre: lustre-OST0000: new disk, initializing [ 8970.173469] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8970.187508] Lustre: 240514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8970.321794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8975.975399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8975.994616] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8976.165186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8978.206874] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8989.087688] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8994.342682] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8994.546861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8999.315636] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8999.501598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9001.957859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9001.967600] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9001.980236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9001.994130] Lustre: Skipped 1 previous similar message [ 9007.077478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9007.088186] Lustre: Skipped 2 previous similar messages [ 9007.477484] Lustre: server umount lustre-OST0000 complete [ 9011.683908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9011.695510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9011.714887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9017.314708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9017.325752] Lustre: Skipped 3 previous similar messages [ 9018.112220] Lustre: server umount lustre-MDT0000 complete [ 9022.715994] LustreError: 238538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790915253 with bad export cookie 16905439655410048551 [ 9022.720081] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9022.740766] LustreError: 238538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9023.444763] Lustre: server umount lustre-MDT0001 complete [ 9046.319749] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 9060.832211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9061.217410] LustreError: 243453:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9061.236610] LustreError: 243453:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9061.280115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9065.989961] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9066.473496] LustreError: 243454:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9071.586741] LustreError: 243453:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9075.874857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9076.532826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9081.383547] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9085.095121] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9092.561087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9093.047318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9099.727563] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9107.922361] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9108.148743] Lustre: lustre-OST0001: new disk, initializing [ 9108.152138] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9108.157538] Lustre: 245759:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9113.595953] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9115.218954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9115.229725] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9115.321178] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9123.290353] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9135.780743] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9151.369438] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 00:29:41 (1790915381) [ 9156.586319] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9156.588746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9156.611368] Lustre: Skipped 3 previous similar messages [ 9156.614630] Lustre: Skipped 1 previous similar message [ 9160.142120] Lustre: server umount lustre-MDT0000 complete [ 9161.697942] LustreError: 244456:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9161.734070] LustreError: 244456:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9163.964316] LustreError: 243434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790915394 with bad export cookie 16905439655410049433 [ 9163.973707] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9164.300929] Lustre: server umount lustre-MDT0001 complete [ 9179.030355] Lustre: server umount lustre-OST0000 complete [ 9182.175883] Lustre: 236761:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790915397/real 1790915397] req@ffff8bc5829fea00 x1877910601451136/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790915413 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9182.219172] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9182.245399] Lustre: Skipped 1 previous similar message [ 9182.435832] Lustre: server umount lustre-OST0001 complete [ 9189.370450] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 9199.686952] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 9235.969846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9236.388604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9236.434189] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9236.522820] Lustre: lustre-MDT0000: new disk, initializing [ 9236.631821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9236.639916] Lustre: Skipped 1 previous similar message [ 9236.660899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9241.488337] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9254.455256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9254.540615] Lustre: 249708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9254.577069] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9254.580733] Lustre: Skipped 1 previous similar message [ 9254.635735] Lustre: lustre-MDT0001: new disk, initializing [ 9254.751471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9254.763032] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9261.432859] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9273.369205] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9280.455421] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9289.224309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9289.672916] Lustre: lustre-OST0000: new disk, initializing [ 9289.681595] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9289.688861] Lustre: 251677:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9289.786476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9289.796283] Lustre: Skipped 1 previous similar message [ 9291.379504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9291.387466] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9291.640530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9297.651653] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9308.098761] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9313.070657] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9313.328644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9318.336889] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9318.537208] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9322.467813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9322.472440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9322.488169] Lustre: Skipped 1 previous similar message [ 9322.510234] Lustre: Skipped 2 previous similar messages [ 9326.650933] Lustre: server umount lustre-OST0000 complete [ 9331.685652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9331.700367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9337.833360] LustreError: 249716:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9337.863207] LustreError: 249716:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9337.888772] Lustre: server umount lustre-MDT0000 complete [ 9343.272395] LustreError: 249700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790915574 with bad export cookie 16905439655410051575 [ 9343.276532] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9343.282375] LustreError: 249700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9344.035738] Lustre: server umount lustre-MDT0001 complete [ 9356.888760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9357.823319] LustreError: 253895:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9357.866936] LustreError: 253895:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9357.995455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9362.940241] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9374.530797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9382.936152] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9393.880792] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9400.123572] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9408.051967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9416.724914] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9425.197780] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9574.369637] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9574.383137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9574.397805] Lustre: Skipped 1 previous similar message [ 9574.404806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9574.416891] Lustre: Skipped 3 previous similar messages [ 9578.737870] Lustre: server umount lustre-OST0000 complete [ 9585.127342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9585.147338] LustreError: Skipped 1 previous similar message [ 9591.307485] Lustre: server umount lustre-MDT0000 complete [ 9595.126345] LustreError: 253876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790915826 with bad export cookie 16905439655410052457 [ 9595.137035] LustreError: 253876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9595.145152] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9595.701573] Lustre: server umount lustre-MDT0001 complete [ 9700.128830] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 9703.248878] Key type lgssc unregistered [ 9703.586768] LNet: 257512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9703.593755] LNetError: 257512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9703.609499] LNet: Removed LNI 192.168.203.153@tcp [ 9704.876799] Key type .llcrypt unregistered [ 9704.881981] Key type ._llcrypt unregistered [ 9733.481867] Key type ._llcrypt registered [ 9733.484660] Key type .llcrypt registered [ 9733.581061] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 9752.394531] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 9753.680525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9753.963389] alg: No test for adler32 (adler32-zlib) [ 9755.366614] Lustre: Lustre: Build Version: 2.17.58_63_g006d100 [ 9755.647566] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 9757.399934] Key type lgssc registered [ 9759.131867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9809.815093] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 9820.400413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9820.451789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9821.660317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9821.692714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9821.790615] Lustre: lustre-MDT0000: new disk, initializing [ 9821.875875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9821.897798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9826.108430] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9837.337127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9837.464712] Lustre: 261599:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9837.541656] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9837.545343] Lustre: Skipped 1 previous similar message [ 9837.617938] Lustre: lustre-MDT0001: new disk, initializing [ 9837.704776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9837.748570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9837.761221] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9843.208336] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9848.675120] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9856.255117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9856.682181] Lustre: lustre-OST0000: new disk, initializing [ 9856.687357] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9856.700886] Lustre: 263233:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9856.825859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9863.069977] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9866.299194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9866.306155] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9866.424856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9873.625975] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9873.776384] Lustre: lustre-OST0001: new disk, initializing [ 9873.779909] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9873.785615] Lustre: 264101:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9873.871060] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9876.398470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9876.411085] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9879.787902] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9881.596266] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9890.440905] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9898.681787] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9912.294483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9912.303713] Lustre: Skipped 1 previous similar message [ 9912.304275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9912.318384] Lustre: Skipped 3 previous similar messages [ 9917.130028] Lustre: server umount lustre-MDT0000 complete [ 9921.419394] LustreError: 261591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790916152 with bad export cookie 6282445192125673478 [ 9921.421501] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9921.431554] LustreError: 261591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9921.843865] Lustre: server umount lustre-MDT0001 complete [ 9937.326561] Lustre: server umount lustre-OST0000 complete [ 9938.271182] Lustre: 259095:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790916153/real 1790916153] req@ffff8bc590b93b80 x1877911521013632/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790916169 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9938.297256] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9941.836185] Lustre: server umount lustre-OST0001 complete [ 9958.761683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9970.198908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9978.916220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9988.643561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10006.457488] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [10015.079662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10015.100945] Lustre: lustre-MDT0000: reset Object Index mappings [10015.426432] LustreError: 269112:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10015.437342] LustreError: 269112:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10015.503163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10019.390898] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10020.833799] LustreError: 269113:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10025.955088] LustreError: 269112:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10027.032789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10027.050626] Lustre: lustre-MDT0001: reset Object Index mappings [10027.540568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10032.212927] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10035.433089] Lustre: 270253:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10042.576918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10049.136788] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10052.154491] LustreError: 270605:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10057.191454] LustreError: 270608:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10057.527368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10057.761890] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10057.767627] Lustre: Skipped 1 previous similar message [10060.879723] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10060.879723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10063.767464] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10071.778588] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10075.616867] Lustre: 272124:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10081.258268] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10081.268719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10081.271747] Lustre: Skipped 2 previous similar messages [10081.285781] Lustre: Skipped 3 previous similar messages [10082.867658] Lustre: server umount lustre-MDT0000 complete [10086.236448] LustreError: 269094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790916317 with bad export cookie 6282445192125680268 [10086.240995] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10086.256207] LustreError: 269094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10086.368527] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10086.369955] LustreError: 270906:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10086.379476] Lustre: Skipped 1 previous similar message [10086.382128] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10086.437452] LustreError: 270906:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10086.646153] Lustre: server umount lustre-MDT0001 complete [10092.256429] Lustre: server umount lustre-OST0000 complete [10095.569139] Lustre: server umount lustre-OST0001 complete [10111.355176] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 00:45:41 (1790916341) [10124.070943] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [10134.070576] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [10165.852340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10166.088894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10166.115321] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10166.177743] Lustre: lustre-MDT0000: new disk, initializing [10166.244357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10166.259440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10170.547856] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10179.666405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10179.797822] Lustre: 276377:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10179.858992] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10179.862020] Lustre: Skipped 1 previous similar message [10179.940554] Lustre: lustre-MDT0001: new disk, initializing [10180.087100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10180.099862] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10184.411946] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10193.549614] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10198.565682] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10205.401062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10205.688924] Lustre: lustre-OST0000: new disk, initializing [10205.704312] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10205.719618] Lustre: 278345:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10207.546381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10207.576057] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10207.660063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10212.583552] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10221.496744] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10225.854911] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10226.022303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10230.204652] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10230.400132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10233.314712] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10233.318905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10233.327624] LustreError: Skipped 1 previous similar message [10233.329752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10233.343586] Lustre: Skipped 2 previous similar messages [10233.354363] Lustre: Skipped 2 previous similar messages [10238.502755] Lustre: server umount lustre-OST0000 complete [10243.553572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10243.555445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10243.572802] Lustre: Skipped 1 previous similar message [10247.872816] Lustre: server umount lustre-MDT0000 complete [10248.673226] LustreError: 276388:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10248.686824] LustreError: 276388:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10251.551207] LustreError: 276369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790916482 with bad export cookie 6282445192125682361 [10251.554174] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10251.561051] LustreError: 276369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10251.960699] Lustre: server umount lustre-MDT0001 complete [10270.391532] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [10280.521810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10280.819994] LustreError: 281280:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10280.865879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10280.872071] Lustre: Skipped 2 previous similar messages [10285.945700] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10295.677992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10301.348775] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10304.313150] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10310.796876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10317.337207] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10326.090569] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10326.282226] Lustre: lustre-OST0001: new disk, initializing [10326.289252] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10326.299369] Lustre: 283587:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10328.152661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10328.162166] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10328.281127] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10333.012974] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10341.872181] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10345.434593] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10350.462752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10350.512543] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10362.224721] Lustre: server umount lustre-MDT0000 complete [10362.849587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10362.855166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10362.863715] LustreError: 281280:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10362.873503] LustreError: 281280:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10365.535689] LustreError: 281260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790916596 with bad export cookie 6282445192125683243 [10365.537449] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10365.545428] LustreError: 281260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10365.839454] Lustre: server umount lustre-MDT0001 complete [10379.909086] Lustre: server umount lustre-OST0000 complete [10393.559997] Lustre: server umount lustre-OST0001 complete [10412.767603] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [10425.585658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10426.258986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10426.262563] Lustre: Skipped 3 previous similar messages [10430.671145] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10439.286158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10444.741606] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10447.377870] Lustre: 288071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10454.082925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10460.025838] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10467.045781] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10471.457607] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10471.490853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10473.107772] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10480.439381] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10484.199705] Lustre: 289949:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10489.037337] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10489.168195] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10501.064272] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 00:52:10 (1790916730) [10516.451520] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10516.456737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10516.476737] Lustre: Skipped 3 previous similar messages [10516.485633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10516.492381] Lustre: Skipped 1 previous similar message [10522.595788] LustreError: 286931:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10522.616218] LustreError: 286931:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [10522.627339] Lustre: server umount lustre-MDT0000 complete [10525.864546] LustreError: 286912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790916756 with bad export cookie 6282445192125689662 [10525.873740] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10532.284612] Lustre: server umount lustre-MDT0001 complete [10541.688267] Lustre: server umount lustre-OST0000 complete [10544.795380] Lustre: server umount lustre-OST0001 complete [10551.350981] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [10561.517348] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [10613.104082] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [10623.056401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10623.323835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10623.345618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10623.404702] Lustre: lustre-MDT0000: new disk, initializing [10623.461547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10628.613550] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10638.836857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10639.003872] Lustre: 294476:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10639.337672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10644.517267] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10655.926597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10656.205810] Lustre: 296108:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10657.921210] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10662.919868] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10673.958568] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10674.099836] Lustre: 296979:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10675.942569] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10675.958445] Lustre: Skipped 1 previous similar message [10676.025081] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10680.477035] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10690.045694] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10693.560732] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10693.565593] Lustre: Skipped 1 previous similar message [10706.150291] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 00:55:36 (1790916936) [10710.281904] Lustre: server umount lustre-MDT0000 complete [10711.010997] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10711.017608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10711.027209] Lustre: Skipped 5 previous similar messages [10713.421425] LustreError: 294468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790916944 with bad export cookie 6282445192125697586 [10713.431665] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10713.438717] LustreError: 294468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10713.684461] Lustre: server umount lustre-MDT0001 complete [10727.643166] Lustre: server umount lustre-OST0000 complete [10741.771464] Lustre: server umount lustre-OST0001 complete [10749.954282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10750.406261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10750.412286] Lustre: Skipped 7 previous similar messages [10754.698957] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10763.684208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10768.183368] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10776.970725] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10783.021457] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10790.080559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10795.701455] LustreError: 301052:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10795.737395] LustreError: 301052:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [10796.613787] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10803.858426] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10823.317912] loop1: detected capacity change from 0 to 204800000 [10823.542833] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10823.554433] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10823.571912] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10828.073718] loop1: detected capacity change from 0 to 204800000 [10833.310474] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10833.427669] LustreError: 299213:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs29585: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10833.450248] LustreError: 299213:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs29585 [10833.455272] LustreError: 299213:0:(mgs_handler.c:517:mgs_target_reg()) fs29585-MDT0001: Error checking target index: rc = -13 [10833.465484] LustreError: fs29585-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10833.473863] LustreError: 302942:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10833.483309] LustreError: 302942:0:(tgt_mount.c:2128:server_put_super()) no obd fs29585-MDT0001 [10833.495133] LustreError: 302942:0:(tgt_mount.c:116:server_deregister_mount()) fs29585-MDT0001 not registered [10833.532980] Lustre: server umount fs29585-MDT0001 complete [10833.537988] LustreError: 302942:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10840.803761] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10840.922117] Lustre: Setting parameter fs29585-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs29585-MDT0001 [10840.976486] Lustre: srv-fs29585-MDT0001: No data found on store. Initialize space. [10840.980299] Lustre: Skipped 4 previous similar messages [10841.039390] Lustre: fs29585-MDT0001: new disk, initializing [10841.042229] Lustre: Skipped 3 previous similar messages [10841.172969] Lustre: srv-fs29585-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10841.183834] Lustre: 303409:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29585-MDT0001-mdtlov: can't create new llog: rc = -115 [10841.195603] LustreError: 303409:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29585-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10844.786341] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10850.461691] Lustre: server umount fs29585-MDT0001 complete [10861.852272] loop1: detected capacity change from 0 to 204800000 [10866.560691] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10866.894698] Lustre: 304975:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29585-MDT0001-mdtlov: can't create new llog: rc = -115 [10866.907222] LustreError: 304975:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29585-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10870.646515] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10874.663493] Lustre: server umount fs29585-MDT0001 complete [10875.880690] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10884.066326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10884.088611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10884.094851] Lustre: Skipped 5 previous similar messages [10884.117470] Lustre: Skipped 10 previous similar messages [10889.220320] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10889.833697] Lustre: server umount lustre-MDT0000 complete [10898.012565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10898.177687] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10902.542280] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10903.536538] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10909.638138] loop1: detected capacity change from 0 to 204800000 [10915.255976] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10915.851175] Lustre: 307425:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29585-MDT0001-mdtlov: can't create new llog: rc = -115 [10915.888770] LustreError: 307425:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29585-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10920.769761] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10926.098402] Lustre: server umount fs29585-MDT0001 complete [10949.609289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10949.622905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10949.628980] Lustre: Skipped 4 previous similar messages [10953.858513] Lustre: server umount lustre-OST0000 complete [10963.701361] Lustre: server umount lustre-MDT0000 complete [10967.765701] LustreError: 299205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790917198 with bad export cookie 6282445192125700267 [10967.771604] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10967.783704] LustreError: 299205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10968.516598] Lustre: server umount lustre-MDT0001 complete [10982.566206] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [10986.202856] Key type lgssc unregistered [10986.548710] LNet: 309768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10986.562520] LNetError: 309768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10986.584198] LNet: Removed LNI 192.168.203.153@tcp [10987.605226] Key type .llcrypt unregistered [10987.610659] Key type ._llcrypt unregistered [11007.264678] Key type ._llcrypt registered [11007.266861] Key type .llcrypt registered [11007.368768] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [11008.360753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11008.401115] alg: No test for adler32 (adler32-zlib) [11009.569749] Lustre: Lustre: Build Version: 2.17.58_63_g006d100 [11009.933444] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [11011.671369] Key type lgssc registered [11012.949805] Lustre: Echo OBD driver; http://www.lustre.org/ [11020.549437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11020.571249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11022.100671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11027.298854] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11032.551428] LustreError: 311053:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11032.563012] LustreError: 311053:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11036.780398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11037.246956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11042.175937] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11049.538229] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11055.522972] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11061.872700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11062.338555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11063.345354] LustreError: 312895:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11067.432467] LustreError: 312897:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11067.453934] LustreError: 312897:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11067.790672] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11072.496023] LustreError: 312896:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11072.509813] LustreError: 312896:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11075.000290] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11076.643723] LustreError: 312895:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11086.900113] LustreError: 312895:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11086.925694] LustreError: 312895:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11090.819822] loop1: detected capacity change from 0 to 204800000 [11091.009422] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11091.022977] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11091.047305] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11096.135810] loop1: detected capacity change from 0 to 204800000 [11101.131162] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11101.310135] Lustre: fs15502-OST0000: new disk, initializing [11101.313417] Lustre: srv-fs15502-OST0000: No data found on store. Initialize space. [11101.318563] Lustre: 314802:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11101.379481] Lustre: fs15502-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11103.209948] LustreError: 312976:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11103.249091] LustreError: 312976:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [11107.811750] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11116.247729] Lustre: server umount fs15502-OST0000 complete [11129.049412] loop1: detected capacity change from 0 to 204800000 [11136.228849] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11136.398764] Lustre: fs15502-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11138.104081] LustreError: 312897:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11138.123518] LustreError: 312897:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 18 previous similar messages [11144.481344] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11149.284989] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11153.790174] loop2: detected capacity change from 0 to 204800000 [11154.046834] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11154.058921] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11154.071255] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11159.641566] loop2: detected capacity change from 0 to 204800000 [11165.833628] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11165.955786] LustreError: fs15502-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11165.982579] LustreError: fs15502-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11165.994964] LustreError: 317844:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11166.027478] LustreError: 317844:0:(tgt_mount.c:2128:server_put_super()) no obd fs15502-OST0001 [11166.037533] LustreError: 317844:0:(tgt_mount.c:116:server_deregister_mount()) fs15502-OST0001 not registered [11166.107882] Lustre: server umount fs15502-OST0001 complete [11166.115950] LustreError: 317844:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11176.571224] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11176.821483] Lustre: fs15502-OST0001: new disk, initializing [11176.828606] Lustre: srv-fs15502-OST0001: No data found on store. Initialize space. [11176.839949] Lustre: 318395:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11176.908254] Lustre: fs15502-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11183.573483] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11192.461557] Lustre: server umount fs15502-OST0000 complete [11202.041785] Lustre: server umount fs15502-OST0001 complete [11203.561933] LustreError: 312896:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11203.579620] LustreError: 312896:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 39 previous similar messages [11214.232993] loop1: detected capacity change from 0 to 204800000 [11220.306113] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11220.632902] Lustre: fs15502-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11227.642722] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11233.907969] loop2: detected capacity change from 0 to 204800000 [11239.675297] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11245.902649] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11255.014528] Lustre: server umount fs15502-OST0000 complete [11265.064681] Lustre: server umount fs15502-OST0001 complete [11278.309555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11278.323521] LustreError: Skipped 1 previous similar message [11278.327165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11278.337870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11280.361220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11280.365144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11280.374441] Lustre: Skipped 1 previous similar message [11283.754996] Lustre: server umount lustre-OST0000 complete [11288.035320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11288.055344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11288.076748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11290.592545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11290.597804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11290.631302] Lustre: Skipped 1 previous similar message [11294.013599] Lustre: server umount lustre-MDT0000 complete [11298.468739] LustreError: 311035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790917529 with bad export cookie 13456352612878901216 [11298.479390] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11298.483975] LustreError: 311035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11299.275795] Lustre: server umount lustre-MDT0001 complete [11312.077325] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [11314.826802] Key type lgssc unregistered [11315.144035] LNet: 324318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11315.156754] LNetError: 324318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11315.180977] LNet: Removed LNI 192.168.203.153@tcp [11316.056242] Key type .llcrypt unregistered [11316.061262] Key type ._llcrypt unregistered [11346.461368] Key type ._llcrypt registered [11346.468824] Key type .llcrypt registered [11346.580523] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [11360.936470] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [11361.900430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11362.029156] alg: No test for adler32 (adler32-zlib) [11363.100699] Lustre: Lustre: Build Version: 2.17.58_63_g006d100 [11363.330736] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [11365.015371] Key type lgssc registered [11366.226211] Lustre: Echo OBD driver; http://www.lustre.org/ [11405.052503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11405.097500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11406.385861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11406.440852] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11406.559543] Lustre: lustre-MDT0000: new disk, initializing [11406.733756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11406.751053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11412.574156] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11423.932508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11424.194379] Lustre: 328193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11424.310751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11424.320460] Lustre: Skipped 1 previous similar message [11424.566590] Lustre: lustre-MDT0001: new disk, initializing [11424.723860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11424.784940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11424.801767] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11430.175051] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11443.010799] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11449.700945] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11458.791726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11459.149307] Lustre: lustre-OST0000: new disk, initializing [11459.154698] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11459.163447] Lustre: 330163:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11459.288243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11464.786519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11464.795280] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11464.957655] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11466.464791] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11476.812132] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11479.458733] Lustre: Mounted lustre-client - version 2.17.58_63_g006d100 [11480.429175] Lustre: Unmounted lustre-client [11490.787218] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11490.793412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11490.797338] LustreError: Skipped 1 previous similar message [11490.799947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11490.827260] Lustre: Skipped 1 previous similar message [11494.887836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11494.899498] Lustre: Skipped 2 previous similar messages [11496.765759] Lustre: server umount lustre-OST0000 complete [11501.536870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11501.553676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11501.569154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11505.133375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11505.144383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11505.167030] Lustre: Skipped 1 previous similar message [11506.896637] Lustre: server umount lustre-MDT0000 complete [11510.704334] LustreError: 328185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790917741 with bad export cookie 14514157249510547215 [11510.709409] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11510.722518] LustreError: 328185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11511.116030] Lustre: server umount lustre-MDT0001 complete [11520.526174] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [11523.164941] Key type lgssc unregistered [11532.320356] LNet: 1 local NIs in recovery (showing 1): 192.168.203.153@tcp [11532.640231] LNet: 332324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11532.653694] LNetError: 332324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11609.521653] LNet: Removed LNI 192.168.203.153@tcp [11610.528996] Key type .llcrypt unregistered [11610.532094] Key type ._llcrypt unregistered [11630.477521] Key type ._llcrypt registered [11630.480187] Key type .llcrypt registered [11630.624694] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [11646.796597] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [11648.097352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11648.307185] alg: No test for adler32 (adler32-zlib) [11649.379824] Lustre: Lustre: Build Version: 2.17.58_63_g006d100 [11649.652666] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [11651.455326] Key type lgssc registered [11652.929479] Lustre: Echo OBD driver; http://www.lustre.org/ [11682.971852] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 01:11:53 (1790917913) [11690.584879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11690.621074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11691.902161] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11691.939571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11692.046180] Lustre: lustre-MDT0000: new disk, initializing [11692.127868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11692.159484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11697.781589] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11709.320989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11709.479537] Lustre: 335564:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11709.568994] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11709.573500] Lustre: Skipped 1 previous similar message [11709.687353] Lustre: lustre-MDT0001: new disk, initializing [11709.813690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11709.891503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11709.920909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11714.344646] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11725.120041] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11732.979857] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11740.959170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11741.413767] Lustre: lustre-OST0000: new disk, initializing [11741.419733] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11741.437243] Lustre: 337536:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11741.561708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11750.564243] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11750.957837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11750.974548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11751.046976] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11761.499351] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11781.602681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11781.606643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11781.642935] Lustre: Skipped 1 previous similar message [11781.667736] Lustre: Skipped 1 previous similar message [11786.099297] Lustre: server umount lustre-OST0000 complete [11791.852486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11791.876039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11791.878221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11791.891364] Lustre: Skipped 1 previous similar message [11796.733624] Lustre: server umount lustre-MDT0000 complete [11800.655858] LustreError: 335558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790918031 with bad export cookie 5165058183812580239 [11800.659719] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11800.687474] LustreError: 335558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11801.280922] Lustre: server umount lustre-MDT0001 complete [11816.381186] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [11820.816681] Key type lgssc unregistered [11821.349337] LNet: 339443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11821.369335] LNetError: 339443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11822.447716] LNet: Removed LNI 192.168.203.153@tcp [11823.883673] Key type .llcrypt unregistered [11823.885778] Key type ._llcrypt unregistered [11847.190246] Key type ._llcrypt registered [11847.195892] Key type .llcrypt registered [11847.291211] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [11848.464508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11848.509784] alg: No test for adler32 (adler32-zlib) [11849.750403] Lustre: Lustre: Build Version: 2.17.58_63_g006d100 [11850.007614] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [11851.671279] Key type lgssc registered [11853.061362] Lustre: Echo OBD driver; http://www.lustre.org/ [11861.527390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11861.545931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11862.940304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11867.692409] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11872.228420] LustreError: 340733:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11872.247962] LustreError: 340733:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11876.321875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11876.601673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11880.980585] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11888.603876] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11895.235578] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11902.718987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11903.103489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11910.427455] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11918.429072] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11933.344590] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11935.123620] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11941.987477] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.53@tcp (stopping) [11942.380518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11942.409938] Lustre: Skipped 2 previous similar messages [11946.034495] Lustre: server umount lustre-MDT0000 complete [11947.060919] LustreError: 343179:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11950.686788] LustreError: 340715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790918181 with bad export cookie 4320701375779773599 [11950.713851] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11952.179864] LustreError: 343369:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11952.208420] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.53@tcp (stopping) [11952.211810] LustreError: 343369:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11952.238113] Lustre: Skipped 3 previous similar messages [11952.613099] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11955.618798] LustreError: 343179:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11955.626430] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11955.658916] LustreError: 343179:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11955.676407] Lustre: Skipped 1 previous similar message [11957.508658] Lustre: server umount lustre-MDT0001 complete [11962.366212] Lustre: server umount lustre-OST0000 complete [11971.164029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11975.720402] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11978.473846] Lustre: 344915:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.153@tcp [11980.440129] Lustre: server umount MGS complete [11988.631420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11989.091086] LustreError: 345487:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11989.106960] LustreError: 345487:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11989.147758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11993.668199] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11999.720044] LustreError: 345488:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11999.740987] LustreError: 345488:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12004.523405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12004.833755] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12005.176261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12011.365068] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [12020.057809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12020.390185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12027.077351] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [12035.363670] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12037.150412] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12042.536843] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12044.333594] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12049.897671] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12049.908051] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12049.947900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12050.846603] Lustre: server umount lustre-OST0000 complete [12055.521390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12055.533139] Lustre: Skipped 2 previous similar messages [12060.641366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12060.649601] Lustre: Skipped 5 previous similar messages [12061.004202] Lustre: server umount lustre-MDT0000 complete [12064.856939] LustreError: 345469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790918295 with bad export cookie 4320701375779774439 [12064.865719] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12064.868445] LustreError: 345469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [12065.308748] Lustre: server umount lustre-MDT0001 complete [12076.572437] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [12079.459496] Key type lgssc unregistered [12079.692213] LNet: 348923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12079.705308] LNetError: 348923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12079.724827] LNet: Removed LNI 192.168.203.153@tcp [12080.596213] Key type .llcrypt unregistered [12080.598829] Key type ._llcrypt unregistered [12099.005759] Key type ._llcrypt registered [12099.010737] Key type .llcrypt registered [12099.101962] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [12117.472408] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [12118.971176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12119.023932] alg: No test for adler32 (adler32-zlib) [12120.204774] Lustre: Lustre: Build Version: 2.17.58_63_g006d100 [12120.439972] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [12122.178119] Key type lgssc registered [12124.449213] Lustre: Echo OBD driver; http://www.lustre.org/ [12166.381773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12166.418984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12167.685394] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12167.711203] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12167.776597] Lustre: lustre-MDT0000: new disk, initializing [12167.827696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12167.841171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12172.012876] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [12181.380744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12181.482274] Lustre: 352100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12181.543524] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12181.547740] Lustre: Skipped 1 previous similar message [12181.645325] Lustre: lustre-MDT0001: new disk, initializing [12181.719288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12181.750838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12181.762308] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12185.852748] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [12196.877453] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12203.189670] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12210.110361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12210.534887] Lustre: lustre-OST0000: new disk, initializing [12210.541252] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12210.550340] Lustre: 354072:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12210.634767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12218.686330] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [12219.492358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12219.502666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12219.666368] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12228.749247] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12234.223861] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12234.520558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12239.387239] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12239.543811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12245.479689] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12245.483158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12245.495644] LustreError: Skipped 1 previous similar message [12245.499099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12245.522521] Lustre: Skipped 1 previous similar message [12248.133184] Lustre: server umount lustre-OST0000 complete [12253.665930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12253.675935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12253.715709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12253.734588] Lustre: Skipped 1 previous similar message [12255.715494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12255.719476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12259.100834] Lustre: server umount lustre-MDT0000 complete [12263.355866] LustreError: 352092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790918494 with bad export cookie 12875419998831577919 [12263.365608] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12263.379305] LustreError: 352092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12263.886842] Lustre: server umount lustre-MDT0001 complete [12274.243844] Lustre: DEBUG MARKER: == conf-sanity test 163: nolock mount option is ignored, not honoured ========================================================== 01:21:44 (1790918504) [12281.675474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12282.100391] LustreError: 356485:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12282.133163] LustreError: 356485:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12282.209174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12287.461278] LustreError: 356486:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12287.486236] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [12292.582716] LustreError: 356485:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12295.712767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12296.079792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12300.667602] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [12307.988973] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12314.675453] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12322.744130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12323.078578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12330.013766] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [12339.682984] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12367.331159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12367.346909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12367.353339] Lustre: Skipped 1 previous similar message [12367.370017] Lustre: Skipped 2 previous similar messages [12372.478083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12372.487245] Lustre: Skipped 1 previous similar message [12373.448582] Lustre: server umount lustre-OST0000 complete [12377.579792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12382.691662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12382.705369] Lustre: Skipped 3 previous similar messages [12384.011470] Lustre: server umount lustre-MDT0000 complete [12387.811674] LustreError: 356486:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12387.838871] LustreError: 356486:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12388.425809] LustreError: 356466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790918619 with bad export cookie 12875419998831578801 [12388.440668] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12388.445463] LustreError: 356466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12388.958157] Lustre: server umount lustre-MDT0001 complete [12400.998539] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [12404.224856] Key type lgssc unregistered [12404.645301] LNet: 360188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12404.665988] LNetError: 360188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12405.749970] LNet: Removed LNI 192.168.203.153@tcp [12406.933501] Key type .llcrypt unregistered [12406.947862] Key type ._llcrypt unregistered [12431.289935] Key type ._llcrypt registered [12431.292932] Key type .llcrypt registered [12431.468786] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [12449.179708] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [12450.278769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12450.406616] alg: No test for adler32 (adler32-zlib) [12451.533297] Lustre: Lustre: Build Version: 2.17.58_63_g006d100 [12451.744595] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [12453.391276] Key type lgssc registered [12454.725064] Lustre: Echo OBD driver; http://www.lustre.org/ [12499.710024] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12499.730489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12501.124312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12501.206127] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12501.381696] Lustre: lustre-MDT0000: new disk, initializing [12501.439495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12501.449906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12506.842651] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [12519.348782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12519.506381] Lustre: 363417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12519.556497] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12519.560131] Lustre: Skipped 1 previous similar message [12519.619817] Lustre: lustre-MDT0001: new disk, initializing [12519.707092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12519.759394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12519.778494] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12525.102648] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [12536.131242] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12543.186817] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12553.684885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12554.170735] Lustre: lustre-OST0000: new disk, initializing [12554.180828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12554.196880] Lustre: 365386:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12554.395935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12562.563544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12562.575941] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12562.712023] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12564.088797] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [12574.079551] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12614.118881] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12614.129818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12614.131186] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12614.134189] Lustre: Skipped 1 previous similar message [12614.154294] Lustre: Skipped 1 previous similar message [12619.235383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12619.254827] Lustre: Skipped 1 previous similar message [12619.562686] Lustre: server umount lustre-OST0000 complete [12624.353146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12624.367815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12624.369317] Lustre: Skipped 1 previous similar message [12624.381737] Lustre: Skipped 1 previous similar message [12629.472956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12629.482317] Lustre: Skipped 1 previous similar message [12629.920688] Lustre: server umount lustre-MDT0000 complete [12634.929047] LustreError: 363407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790918865 with bad export cookie 4619340174712466903 [12634.938737] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12634.944602] LustreError: 363407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12635.608931] Lustre: server umount lustre-MDT0001 complete [12647.240705] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [12649.989282] Key type lgssc unregistered [12650.283832] LNet: 367564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12650.295554] LNetError: 367564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12651.368981] LNet: Removed LNI 192.168.203.153@tcp [12652.314263] Key type .llcrypt unregistered [12652.316305] Key type ._llcrypt unregistered [12743.039701] Key type ._llcrypt registered [12743.042159] Key type .llcrypt registered [12743.160305] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [12743.469556] Key type .llcrypt unregistered [12743.475197] Key type ._llcrypt unregistered [12818.725150] Key type ._llcrypt registered [12818.731076] Key type .llcrypt registered [12818.868557] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [12819.198453] Key type .llcrypt unregistered [12819.203969] Key type ._llcrypt unregistered [12852.253271] Key type ._llcrypt registered [12852.263233] Key type .llcrypt registered [12852.392703] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [12852.614703] Key type .llcrypt unregistered [12852.617818] Key type ._llcrypt unregistered [12879.280705] Key type ._llcrypt registered [12879.284672] Key type .llcrypt registered [12879.477333] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [12879.838404] Key type .llcrypt unregistered [12879.840922] Key type ._llcrypt unregistered [12953.495507] Key type ._llcrypt registered [12953.499235] Key type .llcrypt registered [12953.576216] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [12953.817401] Key type .llcrypt unregistered [12953.820630] Key type ._llcrypt unregistered [12984.933671] Key type ._llcrypt registered [12984.936849] Key type .llcrypt registered [12985.052220] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [12985.347285] Key type .llcrypt unregistered [12985.350723] Key type ._llcrypt unregistered [13013.018970] Key type ._llcrypt registered [13013.027358] Key type .llcrypt registered [13013.183050] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [13013.486398] Key type .llcrypt unregistered [13013.490700] Key type ._llcrypt unregistered [13043.592163] Key type ._llcrypt registered [13043.596681] Key type .llcrypt registered [13043.817197] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [13044.235611] Key type .llcrypt unregistered [13044.239739] Key type ._llcrypt unregistered [13074.546233] Key type ._llcrypt registered [13074.550518] Key type .llcrypt registered [13074.628385] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [13074.838330] Key type .llcrypt unregistered [13074.843414] Key type ._llcrypt unregistered [13107.693717] Key type ._llcrypt registered [13107.696549] Key type .llcrypt registered [13107.777190] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [13108.074350] Key type .llcrypt unregistered [13108.078472] Key type ._llcrypt unregistered [13163.323699] Key type ._llcrypt registered [13163.326140] Key type .llcrypt registered [13163.414996] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [13163.694258] Key type .llcrypt unregistered [13163.698623] Key type ._llcrypt unregistered [13191.717944] Key type ._llcrypt registered [13191.719677] Key type .llcrypt registered [13191.857645] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [13192.115693] Key type .llcrypt unregistered [13192.117566] Key type ._llcrypt unregistered [13217.994206] Key type ._llcrypt registered [13217.996016] Key type .llcrypt registered [13218.125996] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [13218.472329] Key type .llcrypt unregistered [13218.474276] Key type ._llcrypt unregistered [13266.255807] Key type ._llcrypt registered [13266.257886] Key type .llcrypt registered [13266.340867] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [13280.693256] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [13282.059788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13282.087343] alg: No test for adler32 (adler32-zlib) [13283.169980] Lustre: Lustre: Build Version: 2.17.58_63_g006d100 [13283.473410] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [13285.167157] Key type lgssc registered [13286.662602] Lustre: Echo OBD driver; http://www.lustre.org/ [13321.102357] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13321.129037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13322.351980] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13322.397416] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13322.490584] Lustre: tst250fs-MDT0000: new disk, initializing [13322.564305] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13322.583329] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13326.643250] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [13337.082218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13337.233518] Lustre: 387842:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13337.292176] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13337.297208] Lustre: Skipped 1 previous similar message [13337.394899] Lustre: tst250fs-MDT0001: new disk, initializing [13337.517793] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13337.566804] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13337.585264] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13342.318414] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [13346.807866] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13352.854252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13353.102286] Lustre: tst250fs-OST0000: new disk, initializing [13353.108776] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13353.118897] Lustre: 389477:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13353.214114] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13358.671384] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [13360.152288] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13360.160399] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13360.245708] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13368.475385] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13368.710697] Lustre: tst250fs-OST0001: new disk, initializing [13368.717511] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13368.724495] Lustre: 390351:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13368.833122] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13373.975936] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13373.988722] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13374.012374] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000400 [13374.815600] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [13380.488439] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13383.267098] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13383.275192] Lustre: Skipped 2 previous similar messages [13403.108201] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13403.115641] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13403.121625] Lustre: Skipped 3 previous similar messages [13403.132736] Lustre: Skipped 3 previous similar messages [13406.576749] Lustre: server umount tst250fs-MDT0000 complete [13408.238130] LustreError: 388714:0:(ldlm_lib.c:1199:target_handle_connect()) tst250fs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13408.253394] LustreError: 388714:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [13409.705055] LustreError: 387837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790919640 with bad export cookie 542701308540774114 [13409.706584] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13409.719779] LustreError: 387837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13409.947278] Lustre: server umount tst250fs-MDT0001 complete [13423.875718] Lustre: server umount tst250fs-OST0000 complete [13438.029791] Lustre: server umount tst250fs-OST0001 complete [13455.347397] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [13461.780921] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [13490.960206] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 01:42:01 (1790919721) [13492.265743] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13506.071799] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [13516.485755] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [13544.001736] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13272 sec ========= 01:42:54 (1790919774) [13545.582848] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:42:55 (1790919775) === [13548.695474] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:42:58 (1790919778) === [13585.193399] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [13587.488667] Key type lgssc unregistered [13587.776429] LNet: 399485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13587.785712] LNetError: 399485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13587.809458] LNet: Removed LNI 192.168.203.153@tcp [13588.576182] Key type .llcrypt unregistered [13588.579487] Key type ._llcrypt unregistered